Eye-imaging apparatus using diffractive optical elements

ABSTRACT

Examples of eye-imaging apparatus using diffractive optical elements are provided. For example, an optical device comprises a substrate having a proximal surface and a distal surface, a first coupling optical element disposed on one of the proximal and distal surfaces of the substrate, and a second coupling optical element disposed on one of the proximal and distal surfaces of the substrate and offset from the first coupling optical element. The first coupling optical element can be configured to deflect light at an angle to totally internally reflect (TIR) the light between the proximal and distal surfaces and toward the second coupling optical element, and the second coupling optical element can be configured to deflect at an angle out of the substrate. The eye-imaging apparatus can be used in a head-mounted display such as an augmented or virtual reality display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. ProvisionalPatent Application No. 62/474,419, filed Mar. 21, 2017, entitled“EYE-IMAGING APPARATUS USING DIFFRACTIVE OPTICAL ELEMENTS,” the contentsof which are hereby incorporated by reference herein in their entirety.

FIELD

The present disclosure relates to virtual reality and augmented realityimaging and visualization systems and in particular to compact imagingsystems for acquiring images of an eye using coupling optical elementsto direct light to a camera assembly.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” or “augmentedreality” experiences, wherein digitally reproduced images or portionsthereof are presented to a user in a manner wherein they seem to be, ormay be perceived as, real. A virtual reality, or “VR”, scenariotypically involves presentation of digital or virtual image informationwithout transparency to other actual real-world visual input; anaugmented reality, or “AR”, scenario typically involves presentation ofdigital or virtual image information as an augmentation to visualizationof the actual world around the user. A mixed reality, or “MR”, scenariois a type of AR scenario and typically involves virtual objects that areintegrated into, and responsive to, the natural world. For example, inan MR scenario, AR image content may be blocked by or otherwise beperceived as interacting with objects in the real world.

Referring to FIG. 1, an augmented reality scene 10 is depicted wherein auser of an AR technology sees a real-world park-like setting 20featuring people, trees, buildings in the background, and a concreteplatform 30. In addition to these items, the user of the AR technologyalso perceives that he “sees” “virtual content” such as a robot statue40 standing upon the real-world platform 30, and a cartoon-like avatarcharacter 50 flying by which seems to be a personification of a bumblebee, even though these elements 40, 50 do not exist in the real world.Because the human visual perception system is complex, it is challengingto produce an AR technology that facilitates a comfortable,natural-feeling, rich presentation of virtual image elements amongstother virtual or real-world imagery elements.

Systems and methods disclosed herein address various challenges relatedto AR and VR technology.

SUMMARY

Various implementations of methods and apparatus within the scope of theappended claims each have several aspects, no single one of which issolely responsible for the desirable attributes described herein.Without limiting the scope of the appended claims, some prominentfeatures are described herein.

One aspect of the present disclosure provides imaging an object with acamera assembly that does not directly view the object. Accordingly,optical devices according to embodiments described herein are configuredto direct light from an object to an off-axis camera assembly so tocapture an image of the object as if in a direct view position.

In some embodiments, systems, devices, and methods for acquiring animage of an object using an off-axis camera assembly are disclosed. Inone implementation, an optical device is disclosed that may include asubstrate having a proximal surface and a distal surface; a firstcoupling optical element disposed on one of the proximal and distalsurfaces of the substrate; and a second coupling optical elementdisposed on one of the proximal and distal surfaces of the substrate andoffset from the first coupling optical element. The first couplingoptical element may be configured to deflect light at an angle tototally internally reflect (TIR) the light between the proximal anddistal surfaces and toward the second coupling optical element. Thesecond coupling optical element may be configured to deflect light at anangle out of the substrate. In some embodiments, at least one of thefirst and second coupling optical elements include a plurality ofdiffractive features.

In some embodiments, systems, devices, and methods for acquiring animage of an object using an off-axis camera assembly are disclosed. Inone implementation, a head mounted display (HMD) configured to be wornon a head of a user is disclosed that may include a frame; a pair ofoptical elements supported by the frame such that each optical elementof the pair of optical elements is capable of being disposed forward ofan eye of the user; and an imaging system. The imaging system mayinclude a camera assembly mounted to the frame; and an optical devicefor directing light to the camera assembly. The optical device mayinclude a substrate having a proximal surface and a distal surface; afirst coupling optical element disposed on one of the proximal anddistal surfaces of the substrate; and a second coupling optical elementdisposed on one of the proximal and distal surfaces of the substrate andoffset from the first coupling optical element. The first couplingoptical element may be configured to deflect light at an angle to TIRthe light between the proximal and distal surfaces and toward the secondcoupling optical element. The second coupling optical element may beconfigured to deflect light at an angle out of the substrate.

In some embodiments, systems, devices, and methods for acquiring animage of an object using an off-axis camera assembly are disclosed. Inone implementation, an imaging system is disclosed that may include asubstrate having a proximal surface and a distal surface. The substratemay include a first diffractive optical element disposed on one of theproximal and distal surfaces of the substrate, and a second diffractiveoptical element disposed on one of the proximal and distal surfaces ofthe substrate and offset from the first coupling optical element. Thefirst diffractive optical element may be configured to deflect light atan angle to TIR the light between the proximal and distal surfaces andtoward the second coupling optical element. The second diffractiveoptical element may be configured to deflect light incident thereon atan angle out of the substrate. The imaging system may also include acamera assembly to image the light deflected by the second couplingoptical element. In some embodiments, the first and second diffractiveoptical elements comprise at least one of an off-axis diffractiveoptical element (DOE), an off-axis diffraction grating, an off-axisdiffractive optical element (DOE), an off-axis holographic mirror(OAHM), or an off-axis volumetric diffractive optical element (OAVDOE),an off-axis cholesteric liquid crystal diffraction grating (OACLCG), ahot mirror, a prism, or a surface of a decorative lens.

In some embodiments, systems, devices, and methods for acquiring animage of an object using an off-axis camera assembly are disclosed. Themethod may include providing an imaging system in front of an object tobe imaged. The imaging system may a substrate that may include a firstcoupling optical element and a second coupling optical element eachdisposed on one of a proximal surface and a distal surface of thesubstrate and offset from each other. The first coupling optical elementmay be configured to deflect light at an angle to TIR the light betweenthe proximal and distal surfaces and toward the second coupling opticalelement. The second coupling optical element may be configured todeflect light at an angle out of the substrate. The method may alsoinclude capturing light with a camera assembly oriented to receive lightdeflected by the second coupling optical element, and producing anoff-axis image of the object based on the captured light.

In any of the embodiments, the proximal surface and the distal surfaceof the substrate can, but need not, be parallel to each other. Forexample, the substrate may comprise a wedge.

Details of one or more implementations of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims. Neitherthis summary nor the following detailed description purports to defineor limit the scope of the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a user's view of augmented reality (AR) through an ARdevice.

FIG. 2 illustrates an example of a wearable display system.

FIG. 3 illustrates a conventional display system for simulatingthree-dimensional imagery for a user.

FIG. 4 illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes.

FIGS. 5A-5C illustrate relationships between radius of curvature andfocal radius.

FIG. 6 illustrates an example of a waveguide stack for outputting imageinformation to a user.

FIG. 7 illustrates an example of exit beams outputted by a waveguide.

FIG. 8 illustrates an example of a stacked waveguide assembly in whicheach depth plane includes images formed using multiple differentcomponent colors.

FIG. 9A illustrates a cross-sectional side view of an example of a setof stacked waveguides that each includes an in-coupling optical element.

FIG. 9B illustrates a perspective view of an example of the plurality ofstacked waveguides of FIG. 9A.

FIG. 9C illustrates a top-down plan view of an example of the pluralityof stacked waveguides of FIGS. 9A and 9B.

FIGS. 10A & 10B schematically illustrate example imaging systemscomprising a coupling optical element and a camera assembly for trackingan eye.

FIG. 11 schematically illustrates another example imaging systemcomprising multiple coupling optical elements to totally internallyreflect light from an object through a substrate to image the object ata camera assembly.

FIG. 12A schematically illustrates another example imaging systemcomprising multiple coupling optical elements to totally internallyreflect light from an object through a substrate to image the object ata camera assembly.

FIG. 12B is an example image of the object using the imaging system ofFIG. 12A.

FIGS. 13A and 13B schematically illustrate another example imagingsystem comprising multiple coupling optical elements to totallyinternally reflect light from an object through a substrate to image theobject at a camera assembly.

FIGS. 14A-18 schematically illustrate several example arrangements ofimaging systems for imaging an object.

FIG. 19 is a process flow diagram of an example of a method for imagingan object using an off-axis camera.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

DETAILED DESCRIPTION Overview

A head mounted display (HMD) might use information about the state ofthe eyes of the wearer for a variety of purposes. For example, thisinformation can be used for estimating the gaze direction of the wearer,for biometric identification, vision research, evaluate a physiologicalstate of the wearer, etc. However, imaging the eyes can be challenging.The distance between the HMD and the wearer's eyes is short.Furthermore, gaze tracking requires a large field of view (FOV), whilebiometric identification requires a relatively high number of pixels ontarget on the iris. For imaging systems that seek to accomplish both ofthese objectives, these requirements are largely at odds. Furthermore,both problems may be further complicated by occlusion by the eyelids andeyelashes. Some current implementations for tracking eye movement usecameras mounted on the HMD and pointed directly toward the eye tocapture direct images of the eye. However, in order to achieve thedesired FOV and pixel number, the cameras are mounted within thewearer's FOV, thus tend to obstruct and interfere with the wearer'sability to see the surrounding world. Other implementations move thecamera away from obstructing the wearer's view while directly imagingthe eye, which results in imaging the eye from a high angle causingdistortions of the image and reducing the field of view available forimaging the eye.

Embodiments of the imaging systems described herein address some or allof these problems. Various embodiments described herein provideapparatus and systems capable of imaging an eye while permitting thewearer to view the surrounding world. For example, an imaging system cancomprise a substrate disposed along a line of sight between an eye and acamera assembly. The substrate includes one or more coupling opticalelements configured to direct light from the eye into the substrate. Thesubstrate may act as a light-guide (sometimes referred to as awaveguide) to direct light toward the camera assembly. The light maythen exit the substrate and be directed to the camera assembly via oneor more coupling optical elements. The camera assembly receives thelight, thus is able to capture an image (sometimes referred tohereinafter as “direct view image”) of the eye as if in a direct viewposition from a distant position (sometimes referred to herein as“off-axis”).

Some embodiments of the imaging systems described herein provide for asubstrate comprising a first and second coupling optical elementlaterally offset from each other. The substrate includes a surface thatis closest to the eye (sometimes referred to herein as the proximalsurface) and a surface that is furthest from the eye (sometimes referredto as the distal surface). The first and second coupling opticalelements described herein can be disposed on or adjacent to the proximalsurface, on or adjacent to the distal surface, or within the substrate.The first coupling optical element (sometimes referred to herein as anin-coupling optical element) can be configured to deflect light from theeye into the substrate such that the light propagates through thesubstrate by total internal reflection (TIR). The light may be incidenton the second coupling optical element configured to extract the lightand deflect it toward the camera assembly. As used herein, deflect mayrefer to a change in direction of light after interacting something, forexample, an optical component that deflects light may refer toreflection, diffraction, refraction, a change in direction whiletransmitting through the optical component, etc.

In some embodiments, the imaging systems described herein may be aportion of display optics of an HMD (or a lens in a pair of eyeglasses).One or more coupling optical elements may be selected to deflect on afirst range of wavelengths while permitting unhindered propagation of asecond range of wavelengths (for example, a range of wavelengthsdifferent from the first range) through the substrate. The first rangeof wavelengths can be in the infrared (IR), and the second range ofwavelengths can be in the visible. For example, the substrate cancomprise a reflective coupling optical element, which reflects IR lightwhile transmitting visible light. In effect, the imaging system acts asif there were a virtual camera assembly directed back toward thewearer's eye. Thus, virtual camera assembly can image virtual IR lightpropagated from the wearer's eye through the substrate, while visiblelight from the outside world can be transmitted through the substrateand can be perceived by the wearer.

The camera assembly may be configured to view an eye of a wearer, forexample, to capture images of the eye. The camera assembly can bemounted in proximity to the wearer's eye such that the camera assemblydoes not obstruct the wearer's view of the surrounding world or impedethe operation of the HMD. In some embodiments, the camera assembly canbe positioned on a frame of a wearable display system, for example, anear stem or embedded in the eyepiece of the HMD, or below the eye andover the cheek. In some embodiments, a second camera assembly can beused for the wearer's other eye so that each eye can be separatelyimaged. The camera assembly can include an IR digital camera sensitiveto IR radiation.

The camera assembly can be mounted so that it is facing forward (in thedirection of the wearer's vision) or it can be backward facing anddirected toward the eye. In some embodiments, by disposing the cameraassembly nearer the ear of the wearer, the weight of the camera assemblymay also be nearer the ear, and the HMD may be easier to wear ascompared to an HMD where the camera assembly is disposed nearer to thefront of the HMD or in a direct view arrangement. Additionally, byplacing the camera assembly near the wearer's temple, the distance fromthe wearer's eye to the camera assembly is roughly twice as large ascompared to a camera assembly disposed near the front of the HMD. Sincethe depth of field of an image is roughly proportional to this distance,the depth of field for the camera assembly is roughly twice as large ascompared to a direct view camera assembly. A larger depth of field forthe camera assembly can be advantageous for imaging the eye region ofwearers having large or protruding noses, brow ridges, etc. In someembodiments, the position of the camera assembly may be based on thepackaging or design considerations of the HMD. For example, it may beadvantageous to disposed the camera assembly as a backward or forwardfacing in some configurations.

Without subscribing to any particular scientific theory, the embodimentsdescribed herein may include several non-limiting advantages. Severalembodiments are capable of increasing the physical distance between thecamera assembly and the eye, which may facilitate positioning the cameraassembly out of the field of view of the wearer's and therefore notobstructing the wearer's view while permitting capturing of an directview image of the eye. Some of the embodiments described herein also maybe configured to permit eye tracking using larger field of view thanconventional systems thus allowing eye tracking over a wide range ofpositions. The use of IR imaging may facilitate imaging the eye withinterfering with the wearer's ability to see through the substrate andview the environment.

Reference will now be made to the figures, in which like referencenumerals refer to like parts throughout.

Example HMD Device

FIG. 2 illustrates an example of wearable display system 60. The displaysystem 60 includes a display 70, and various mechanical and electronicmodules and systems to support the functioning of that display 70. Thedisplay 70 may be coupled to a frame 80, which is wearable by a displaysystem user or viewer 90 and which is configured to position the display70 in front of the eyes of the user 90. The display 70 may be consideredeyewear in some embodiments. In some embodiments, a speaker 100 iscoupled to the frame 80 and configured to be positioned adjacent the earcanal of the user 90 (in some embodiments, another speaker, not shown,is positioned adjacent the other ear canal of the user to providestereo/shapeable sound control). In some embodiments, the display systemmay also include one or more microphones 110 or other devices to detectsound. In some embodiments, the microphone is configured to allow theuser to provide inputs or commands to the system 60 (e.g., the selectionof voice menu commands, natural language questions, etc.), and/or mayallow audio communication with other persons (e.g., with other users ofsimilar display systems. The microphone may further be configured as aperipheral sensor to collect audio data (e.g., sounds from the userand/or environment). In some embodiments, the display system may alsoinclude a peripheral sensor 120 a, which may be separate from the frame80 and attached to the body of the user 90 (e.g., on the head, torso, anextremity, etc. of the user 90). The peripheral sensor 120 a may beconfigured to acquire data characterizing the physiological state of theuser 90 in some embodiments. For example, the sensor 120 a may be anelectrode.

With continued reference to FIG. 2, the display 70 is operativelycoupled by communications link 130, such as by a wired lead or wirelessconnectivity, to a local data processing module 140 which may be mountedin a variety of configurations, such as fixedly attached to the frame80, fixedly attached to a helmet or hat worn by the user, embedded inheadphones, or otherwise removably attached to the user 90 (e.g., in abackpack-style configuration, in a belt-coupling style configuration).Similarly, the sensor 120 a may be operatively coupled by communicationslink 120 b, e.g., a wired lead or wireless connectivity, to the localprocessor and data module 140. The local processing and data module 140may comprise a hardware processor, as well as digital memory, such asnon-volatile memory (e.g., flash memory or hard disk drives), both ofwhich may be utilized to assist in the processing, caching, and storageof data. The data may include data a) captured from sensors (which maybe, e.g., operatively coupled to the frame 80 or otherwise attached tothe user 90), such as image capture devices (such as, for example,cameras), microphones, inertial measurement units, accelerometers,compasses, GPS units, radio devices, gyros, and/or other sensorsdisclosed herein; and/or b) acquired and/or processed using remoteprocessing module 150 and/or remote data repository 160 (including datarelating to virtual content), possibly for passage to the display 70after such processing or retrieval. The local processing and data module140 may be operatively coupled by communication links 170, 180, such asvia a wired or wireless communication links, to the remote processingmodule 150 and remote data repository 160 such that these remote modules150, 160 are operatively coupled to each other and available asresources to the local processing and data module 140. In someembodiments, the local processing and data module 140 may include one ormore of the image capture devices, microphones, inertial measurementunits, accelerometers, compasses, GPS units, radio devices, and/orgyros. In some other embodiments, one or more of these sensors may beattached to the frame 80, or may be standalone structures thatcommunicate with the local processing and data module 140 by wired orwireless communication pathways.

With continued reference to FIG. 2, in some embodiments, the remoteprocessing module 150 may comprise one or more processors configured toanalyze and process data and/or image information. In some embodiments,the remote data repository 160 may comprise a digital data storagefacility, which may be available through the internet or othernetworking configuration in a “cloud” resource configuration. In someembodiments, the remote data repository 160 may include one or moreremote servers, which provide information, e.g., information forgenerating augmented reality content, to the local processing and datamodule 140 and/or the remote processing module 150. In some embodiments,all data is stored and all computations are performed in the localprocessing and data module, allowing fully autonomous use from a remotemodule.

The perception of an image as being “three-dimensional” or “3-D” may beachieved by providing slightly different presentations of the image toeach eye of the viewer. FIG. 3 illustrates a conventional display systemfor simulating three-dimensional imagery for a user. Two distinct images190, 200—one for each eye 210, 220—are outputted to the user. The images190, 200 are spaced from the eyes 210, 220 by a distance 230 along anoptical or z-axis that is parallel to the line of sight of the viewer.The images 190, 200 are flat and the eyes 210, 220 may focus on theimages by assuming a single accommodated state. Such 3-D display systemsrely on the human visual system to combine the images 190, 200 toprovide a perception of depth and/or scale for the combined image.

It will be appreciated, however, that the human visual system is morecomplicated and providing a realistic perception of depth is morechallenging. For example, many viewers of conventional “3-D” displaysystems find such systems to be uncomfortable or may not perceive asense of depth at all. Without being limited by theory, it is believedthat viewers of an object may perceive the object as being“three-dimensional” due to a combination of vergence and accommodation.Vergence movements (e.g., rotation of the eyes so that the pupils movetoward or away from each other to converge the lines of sight of theeyes to fixate upon an object) of the two eyes relative to each otherare closely associated with focusing (or “accommodation”) of the lensesand pupils of the eyes. Under normal conditions, changing the focus ofthe lenses of the eyes, or accommodating the eyes, to change focus fromone object to another object at a different distance will automaticallycause a matching change in vergence to the same distance, under arelationship known as the “accommodation-vergence reflex,” as well aspupil dilation or constriction. Likewise, a change in vergence willtrigger a matching change in accommodation of lens shape and pupil size,under normal conditions. As noted herein, many stereoscopic or “3-D”display systems display a scene using slightly different presentations(and, so, slightly different images) to each eye such that athree-dimensional perspective is perceived by the human visual system.Such systems are uncomfortable for many viewers, however, since they,among other things, simply provide a different presentation of a scene,but with the eyes viewing all the image information at a singleaccommodated state, and work against the “accommodation-vergencereflex.” Display systems that provide a better match betweenaccommodation and vergence may form more realistic and comfortablesimulations of three-dimensional imagery contributing to increasedduration of wear and in turn compliance to diagnostic and therapyprotocols.

FIG. 4 illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes. With reference toFIG. 4, objects at various distances from eyes 210, 220 on the z-axisare accommodated by the eyes 210, 220 so that those objects are infocus. The eyes 210, 220 assume particular accommodated states to bringinto focus objects at different distances along the z-axis.Consequently, a particular accommodated state may be said to beassociated with a particular one of depth planes 240, which has anassociated focal distance, such that objects or parts of objects in aparticular depth plane are in focus when the eye is in the accommodatedstate for that depth plane. In some embodiments, three-dimensionalimagery may be simulated by providing different presentations of animage for each of the eyes 210, 220, and also by providing differentpresentations of the image corresponding to each of the depth planes.While shown as being separate for clarity of illustration, it will beappreciated that the fields of view of the eyes 210, 220 may overlap,for example, as distance along the z-axis increases. In addition, whileshown as flat for ease of illustration, it will be appreciated that thecontours of a depth plane may be curved in physical space, such that allfeatures in a depth plane are in focus with the eye in a particularaccommodated state.

The distance between an object and the eye 210 or 220 may also changethe amount of divergence of light from that object, as viewed by thateye. FIGS. 5A-5C illustrate relationships between distance and thedivergence of light rays. The distance between the object and the eye210 is represented by, in order of decreasing distance, R1, R2, and R3.As shown in FIGS. 5A-5C, the light rays become more divergent asdistance to the object decreases. As distance increases, the light raysbecome more collimated. Stated another way, it may be said that thelight field produced by a point (the object or a part of the object) hasa spherical wavefront curvature, which is a function of how far away thepoint is from the eye of the user. The curvature increases withdecreasing distance between the object and the eye 210. Consequently, atdifferent depth planes, the degree of divergence of light rays is alsodifferent, with the degree of divergence increasing with decreasingdistance between depth planes and the viewer's eye 210. While only asingle eye 210 is illustrated for clarity of illustration in FIGS. 5A-5Cand other figures herein, it will be appreciated that the discussionsregarding eye 210 may be applied to both eyes 210 and 220 of a viewer.

Without being limited by theory, it is believed that the human eyetypically can interpret a finite number of depth planes to provide depthperception. Consequently, a highly believable simulation of perceiveddepth may be achieved by providing, to the eye, different presentationsof an image corresponding to each of these limited number of depthplanes. The different presentations may be separately focused by theviewer's eyes, thereby helping to provide the user with depth cues basedon the accommodation of the eye required to bring into focus differentimage features for the scene located on different depth plane and/orbased on observing different image features on different depth planesbeing out of focus.

Example of a Waveguide Stack Assembly

FIG. 6 illustrates an example of a waveguide stack for outputting imageinformation to a user. A display system 250 includes a stack ofwaveguides, or stacked waveguide assembly, 260 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 270, 280, 290, 300, 310. In some embodiments, the displaysystem 250 is the system 60 of FIG. 2, with FIG. 6 schematically showingsome parts of that system 60 in greater detail. For example, thewaveguide assembly 260 may be part of the display 70 of FIG. 2. It willbe appreciated that the display system 250 may be considered a lightfield display in some embodiments.

With continued reference to FIG. 6, the waveguide assembly 260 may alsoinclude a plurality of features 320, 330, 340, 350 between thewaveguides. In some embodiments, the features 320, 330, 340, 350 may beone or more lenses. The waveguides 270, 280, 290, 300, 310 and/or theplurality of lenses 320, 330, 340, 350 may be configured to send imageinformation to the eye with various levels of wavefront curvature orlight ray divergence. Each waveguide level may be associated with aparticular depth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 360, 370,380, 390, 400 may function as a source of light for the waveguides andmay be utilized to inject image information into the waveguides 270,280, 290, 300, 310, each of which may be configured, as describedherein, to distribute incoming light across each respective waveguide,for output toward the eye 210. Light exits an output surface 410, 420,430, 440, 450 of the image injection devices 360, 370, 380, 390, 400 andis injected into a corresponding input surface 460, 470, 480, 490, 500of the waveguides 270, 280, 290, 300, 310. In some embodiments, the eachof the input surfaces 460, 470, 480, 490, 500 may be an edge of acorresponding waveguide, or may be part of a major surface of thecorresponding waveguide (that is, one of the waveguide surfaces directlyfacing the world 510 or the viewer's eye 210). In some embodiments, asingle beam of light (e.g. a collimated beam) may be injected into eachwaveguide to output an entire field of cloned collimated beams that aredirected toward the eye 210 at particular angles (and amounts ofdivergence) corresponding to the depth plane associated with aparticular waveguide. In some embodiments, a single one of the imageinjection devices 360, 370, 380, 390, 400 may be associated with andinject light into a plurality (e.g., three) of the waveguides 270, 280,290, 300, 310.

In some embodiments, the image injection devices 360, 370, 380, 390, 400are discrete displays that each produce image information for injectioninto a corresponding waveguide 270, 280, 290, 300, 310, respectively. Insome other embodiments, the image injection devices 360, 370, 380, 390,400 are the output ends of a single multiplexed display which may, e.g.,pipe image information via one or more optical conduits (such as fiberoptic cables) to each of the image injection devices 360, 370, 380, 390,400. It will be appreciated that the image information provided by theimage injection devices 360, 370, 380, 390, 400 may include light ofdifferent wavelengths, or colors (e.g., different component colors, asdiscussed herein).

In some embodiments, the light injected into the waveguides 270, 280,290, 300, 310 is provided by a light projector system 520, whichcomprises a light module 530, which may include a light emitter, such asa light emitting diode (LED). The light from the light module 530 may bedirected to and modified by a light modulator 540, e.g., a spatial lightmodulator, via a beam splitter 550. The light modulator 540 may beconfigured to change the perceived intensity of the light injected intothe waveguides 270, 280, 290, 300, 310. Examples of spatial lightmodulators include liquid crystal displays (LCD) including a liquidcrystal on silicon (LCOS) displays.

In some embodiments, the display system 250 may be a scanning fiberdisplay comprising one or more scanning fibers configured to projectlight in various patterns (e.g., raster scan, spiral scan, Lissajouspatterns, etc.) into one or more waveguides 270, 280, 290, 300, 310 andultimately to the eye 210 of the viewer. In some embodiments, theillustrated image injection devices 360, 370, 380, 390, 400 mayschematically represent a single scanning fiber or a bundle of scanningfibers configured to inject light into one or a plurality of thewaveguides 270, 280, 290, 300, 310. In some other embodiments, theillustrated image injection devices 360, 370, 380, 390, 400 mayschematically represent a plurality of scanning fibers or a plurality ofbundles of scanning fibers, each of which are configured to inject lightinto an associated one of the waveguides 270, 280, 290, 300, 310. Itwill be appreciated that one or more optical fibers may be configured totransmit light from the light module 530 to the one or more waveguides270, 280, 290, 300, and 310. It will be appreciated that one or moreintervening optical structures may be provided between the scanningfiber, or fibers, and the one or more waveguides 270, 280, 290, 300, 310to, e.g., redirect light exiting the scanning fiber into the one or morewaveguides 270, 280, 290, 300, 310.

A controller 560 controls the operation of one or more of the stackedwaveguide assembly 260, including operation of the image injectiondevices 360, 370, 380, 390, 400, the light source 530, and the lightmodulator 540. In some embodiments, the controller 560 is part of thelocal data processing module 140. The controller 560 includesprogramming (e.g., instructions in a non-transitory medium) thatregulates the timing and provision of image information to thewaveguides 270, 280, 290, 300, 310 according to, e.g., any of thevarious schemes disclosed herein. In some embodiments, the controllermay be a single integral device, or a distributed system connected bywired or wireless communication channels. The controller 560 may be partof the processing modules 140 or 150 (FIG. 2) in some embodiments.

With continued reference to FIG. 6, the waveguides 270, 280, 290, 300,310 may be configured to propagate light within each respectivewaveguide by TIR. The waveguides 270, 280, 290, 300, 310 may each beplanar or have another shape (e.g., curved), with major top and bottomsurfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 270, 280,290, 300, 310 may each include out-coupling optical elements 570, 580,590, 600, 610 that are configured to extract light out of a waveguide byredirecting the light, propagating within each respective waveguide, outof the waveguide to output image information to the eye 210. Extractedlight may also be referred to as out-coupled light and the out-couplingoptical elements light may also be referred to light extracting opticalelements. An extracted beam of light may be outputted by the waveguideat locations at which the light propagating in the waveguide strikes alight extracting optical element. The out-coupling optical elements 570,580, 590, 600, 610 may, for example, be gratings, including diffractiveoptical features, as discussed further herein. While illustrateddisposed at the bottom major surfaces of the waveguides 270, 280, 290,300, 310, for ease of description and drawing clarity, in someembodiments, the out-coupling optical elements 570, 580, 590, 600, 610may be disposed at the top and/or bottom major surfaces, and/or may bedisposed directly in the volume of the waveguides 270, 280, 290, 300,310, as discussed further herein. In some embodiments, the out-couplingoptical elements 570, 580, 590, 600, 610 may be formed in a layer ofmaterial that is attached to a transparent substrate to form thewaveguides 270, 280, 290, 300, 310. In some other embodiments, thewaveguides 270, 280, 290, 300, 310 may be a monolithic piece of materialand the out-coupling optical elements 570, 580, 590, 600, 610 may beformed on a surface and/or in the interior of that piece of material.

With continued reference to FIG. 6, as discussed herein, each waveguide270, 280, 290, 300, 310 is configured to output light to form an imagecorresponding to a particular depth plane. For example, the waveguide270 nearest the eye may be configured to deliver collimated light (whichwas injected into such waveguide 270), to the eye 210. The collimatedlight may be representative of the optical infinity focal plane. Thenext waveguide up 280 may be configured to send out collimated lightwhich passes through the first lens 350 (e.g., a negative lens) beforeit can reach the eye 210; such first lens 350 may be configured tocreate a slight convex wavefront curvature so that the eye/braininterprets light coming from that next waveguide up 280 as coming from afirst focal plane closer inward toward the eye 210 from opticalinfinity. Similarly, the third up waveguide 290 passes its output lightthrough both the first 350 and second 340 lenses before reaching the eye210; the combined optical power of the first 350 and second 340 lensesmay be configured to create another incremental amount of wavefrontcurvature so that the eye/brain interprets light coming from the thirdwaveguide 290 as coming from a second focal plane that is even closerinward toward the person from optical infinity than was light from thenext waveguide up 280.

The other waveguide layers 300, 310 and lenses 330, 320 are similarlyconfigured, with the highest waveguide 310 in the stack sending itsoutput through all of the lenses between it and the eye for an aggregatefocal power representative of the closest focal plane to the person. Tocompensate for the stack of lenses 320, 330, 340, 350 whenviewing/interpreting light coming from the world 510 on the other sideof the stacked waveguide assembly 260, a compensating lens layer 620 maybe disposed at the top of the stack to compensate for the aggregatepower of the lens stack 320, 330, 340, 350 below. Such a configurationprovides as many perceived focal planes as there are availablewaveguide/lens pairings. Both the out-coupling optical elements of thewaveguides and the focusing aspects of the lenses may be static (i.e.,not dynamic or electro-active). In some alternative embodiments, eitheror both may be dynamic using electro-active features.

In some embodiments, two or more of the waveguides 270, 280, 290, 300,310 may have the same associated depth plane. For example, multiplewaveguides 270, 280, 290, 300, 310 may be configured to output imagesset to the same depth plane, or multiple subsets of the waveguides 270,280, 290, 300, 310 may be configured to output images set to the sameplurality of depth planes, with one set for each depth plane. This canprovide advantages for forming a tiled image to provide an expandedfield of view at those depth planes.

With continued reference to FIG. 6, the out-coupling optical elements570, 580, 590, 600, 610 may be configured to both redirect light out oftheir respective waveguides and to output this light with theappropriate amount of divergence or collimation for a particular depthplane associated with the waveguide. As a result, waveguides havingdifferent associated depth planes may have different configurations ofout-coupling optical elements 570, 580, 590, 600, 610, which outputlight with a different amount of divergence depending on the associateddepth plane. In some embodiments, the light extracting optical elements570, 580, 590, 600, 610 may be volumetric or surface features, which maybe configured to output light at specific angles. For example, the lightextracting optical elements 570, 580, 590, 600, 610 may be volumeholograms, surface holograms, and/or diffraction gratings. In someembodiments, the features 320, 330, 340, 350 may not be lenses; rather,they may simply be spacers (e.g., cladding layers and/or structures forforming air gaps).

In some embodiments, the out-coupling optical elements 570, 580, 590,600, 610 are diffractive features that form a diffraction pattern, or“diffractive optical element” (also referred to herein as a “DOE”).Preferably, the DOE's have a sufficiently low diffraction efficiency sothat only a portion of the light of the beam is deflected away towardthe eye 210 with each intersection of the DOE, while the rest continuesto move through a waveguide via TIR. The light carrying the imageinformation is thus divided into a number of related exit beams thatexit the waveguide at a multiplicity of locations and the result is afairly uniform pattern of exit emission toward the eye 210 for thisparticular collimated beam bouncing around within a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”states in which they actively diffract, and “off” states in which theydo not significantly diffract. For instance, a switchable DOE maycomprise a layer of polymer dispersed liquid crystal, in whichmicrodroplets comprise a diffraction pattern in a host medium, and therefractive index of the microdroplets may be switched to substantiallymatch the refractive index of the host material (in which case thepattern does not appreciably diffract incident light) or themicrodroplet may be switched to an index that does not match that of thehost medium (in which case the pattern actively diffracts incidentlight).

In some embodiments, a camera assembly 630 (e.g., a digital camera,including visible light and IR light cameras) may be provided to captureimages of the eye 210, parts of the eye 210, or at least a portion ofthe tissue surrounding the eye 210 to, e.g., detect user inputs, extractbiometric information from the eye, estimate and track the gaze of thedirection of the eye, to monitor the physiological state of the user,etc. As used herein, a camera may be any image capture device. In someembodiments, the camera assembly 630 may include an image capture deviceand a light source 632 to project light (e.g., IR or near-IR light) tothe eye, which may then be reflected by the eye and detected by theimage capture device. In some embodiments, the light source 632 includeslight emitting diodes (“LEDs”), emitting in IR or near-IR. While thelight source 632 is illustrated as attached to the camera assembly 630,it will be appreciated that the light source 632 may be disposed inother areas with respect to the camera assembly such that light emittedby the light source is directed to the eye of the wearer (e.g., lightsource 530 described below). In some embodiments, the camera assembly630 may be attached to the frame 80 (FIG. 2) and may be in electricalcommunication with the processing modules 140 or 150, which may processimage information from the camera assembly 630 to make variousdeterminations regarding, e.g., the physiological state of the user, thegaze direction of the wearer, iris identification, etc., as discussedherein. It will be appreciated that information regarding thephysiological state of user may be used to determine the behavioral oremotional state of the user. Examples of such information includemovements of the user or facial expressions of the user. The behavioralor emotional state of the user may then be triangulated with collectedenvironmental or virtual content data so as to determine relationshipsbetween the behavioral or emotional state, physiological state, andenvironmental or virtual content data. In some embodiments, one cameraassembly 630 may be utilized for each eye, to separately monitor eacheye.

With reference now to FIG. 7, an example of exit beams outputted by awaveguide is shown. One waveguide is illustrated, but it will beappreciated that other waveguides in the waveguide assembly 260 (FIG. 6)may function similarly, where the waveguide assembly 260 includesmultiple waveguides. Light 640 is injected into the waveguide 270 at theinput surface 460 of the waveguide 270 and propagates within thewaveguide 270 by TIR. At points where the light 640 impinges on the DOE570, a portion of the light exits the waveguide as exit beams 650. Theexit beams 650 are illustrated as substantially parallel but, asdiscussed herein, they may also be redirected to propagate to the eye210 at an angle (e.g., forming divergent exit beams), depending on thedepth plane associated with the waveguide 270. Substantially parallelexit beams may be indicative of a waveguide with out-coupling opticalelements that out-couple light to form images that appear to be set on adepth plane at a large distance (e.g., optical infinity) from the eye210. Other waveguides or other sets of out-coupling optical elements mayoutput an exit beam pattern that is more divergent, which would requirethe eye 210 to accommodate to a closer distance to bring it into focuson the retina and would be interpreted by the brain as light from adistance closer to the eye 210 than optical infinity.

In some embodiments, a full color image may be formed at each depthplane by overlaying images in each of the component colors, e.g., threeor more component colors. FIG. 8 illustrates an example of a stackedwaveguide assembly in which each depth plane includes images formedusing multiple different component colors. The illustrated embodimentshows depth planes 240 a-240 f, although more or fewer depths are alsocontemplated. Each depth plane may have three or more component colorimages associated with it, including: a first image of a first color, G;a second image of a second color, R; and a third image of a third color,B. Different depth planes are indicated in the figure by differentnumbers for diopters (dpt) following the letters G, R, and B. Just asexamples, the numbers following each of these letters indicate diopters(1/m), or inverse distance of the depth plane from a viewer, and eachbox in the figures represents an individual component color image. Insome embodiments, to account for differences in the eye's focusing oflight of different wavelengths, the exact placement of the depth planesfor different component colors may vary. For example, differentcomponent color images for a given depth plane may be placed on depthplanes corresponding to different distances from the user. Such anarrangement may increase visual acuity and user comfort or may decreasechromatic aberrations.

In some embodiments, light of each component color may be outputted by asingle dedicated waveguide and, consequently, each depth plane may havemultiple waveguides associated with it. In such embodiments, each box inthe figures including the letters G, R, or B may be understood torepresent an individual waveguide, and three waveguides may be providedper depth plane where three component color images are provided perdepth plane. While the waveguides associated with each depth plane areshown adjacent to one another in this drawing for ease of description,it will be appreciated that, in a physical device, the waveguides mayall be arranged in a stack with one waveguide per level. In some otherembodiments, multiple component colors may be outputted by the samewaveguide, such that, e.g., only a single waveguide may be provided perdepth plane.

With continued reference to FIG. 8, in some embodiments, G is the colorgreen, R is the color red, and B is the color blue. In some otherembodiments, other colors associated with other wavelengths of light,including magenta and cyan, may be used in addition to or may replaceone or more of red, green, or blue. In some embodiments, features 320,330, 340, and 350 may be active or passive optical filters configured toblock or selectively pass light from the ambient environment to theviewer's eyes.

It will be appreciated that references to a given color of lightthroughout this disclosure will be understood to encompass light of oneor more wavelengths within a range of wavelengths of light that areperceived by a viewer as being of that given color. For example, redlight may include light of one or more wavelengths in the range of about620-780 nm, green light may include light of one or more wavelengths inthe range of about 492-577 nm, and blue light may include light of oneor more wavelengths in the range of about 435-493 nm.

In some embodiments, the light source 530 (FIG. 6) may be configured toemit light of one or more wavelengths outside the visual perceptionrange of the viewer, for example, IR or ultraviolet wavelengths. IRlight can include light with wavelengths in a range from 700 nm to 10μm. In some embodiments, IR light can include near-IR light withwavelengths in a range from 700 nm to 1.5 μm. In addition, thein-coupling, out-coupling, and other light redirecting structures of thewaveguides of the display 250 may be configured to direct and emit thislight out of the display towards the user's eye 210, e.g., for imagingor user stimulation applications.

With reference now to FIG. 9A, in some embodiments, light impinging on awaveguide may need to be redirected to in-couple the light into thewaveguide. An in-coupling optical element may be used to redirect andin-couple the light into its corresponding waveguide. FIG. 9Aillustrates a cross-sectional side view of an example of a plurality orset 660 of stacked waveguides that each includes an in-coupling opticalelement. The waveguides may each be configured to output light of one ormore different wavelengths, or one or more different ranges ofwavelengths. It will be appreciated that the stack 660 may correspond tothe stack 260 (FIG. 6) and the illustrated waveguides of the stack 660may correspond to part of the plurality of waveguides 270, 280, 290,300, 310, except that light from one or more of the image injectiondevices 360, 370, 380, 390, 400 is injected into the waveguides from aposition that requires light to be redirected for in-coupling.

The illustrated set 660 of stacked waveguides includes waveguides 670,680, and 690. Each waveguide includes an associated in-coupling opticalelement (which may also be referred to as a light input area on thewaveguide), with, e.g., in-coupling optical element 700 disposed on amajor surface (e.g., an upper major surface) of waveguide 670,in-coupling optical element 710 disposed on a major surface (e.g., anupper major surface) of waveguide 680, and in-coupling optical element720 disposed on a major surface (e.g., an upper major surface) ofwaveguide 690. In some embodiments, one or more of the in-couplingoptical elements 700, 710, 720 may be disposed on the bottom majorsurface of the respective waveguide 670, 680, 690 (particularly wherethe one or more in-coupling optical elements are reflective, deflectingoptical elements). As illustrated, the in-coupling optical elements 700,710, 720 may be disposed on the upper major surface of their respectivewaveguide 670, 680, 690 (or the top of the next lower waveguide),particularly where those in-coupling optical elements are transmissive,deflecting optical elements. In some embodiments, the in-couplingoptical elements 700, 710, 720 may be disposed in the body of therespective waveguide 670, 680, 690. In some embodiments, as discussedherein, the in-coupling optical elements 700, 710, 720 are wavelengthselective, such that they selectively redirect one or more wavelengthsof light, while transmitting other wavelengths of light. Whileillustrated on one side or corner of their respective waveguide 670,680, 690, it will be appreciated that the in-coupling optical elements700, 710, 720 may be disposed in other areas of their respectivewaveguide 670, 680, 690 in some embodiments.

As illustrated, the in-coupling optical elements 700, 710, 720 may belaterally offset from one another. In some embodiments, each in-couplingoptical element may be offset such that it receives light without thatlight passing through another in-coupling optical element. For example,each in-coupling optical element 700, 710, 720 may be configured toreceive light from a different image injection device 360, 370, 380,390, and 400 as shown in FIG. 6, and may be separated (e.g., laterallyspaced apart) from other in-coupling optical elements 700, 710, 720 suchthat it substantially does not receive light from the other ones of thein-coupling optical elements 700, 710, 720.

Each waveguide also includes associated light distributing elements,with, e.g., light distributing elements 730 disposed on a major surface(e.g., a top major surface) of waveguide 670, light distributingelements 740 disposed on a major surface (e.g., a top major surface) ofwaveguide 680, and light distributing elements 750 disposed on a majorsurface (e.g., a top major surface) of waveguide 690. In some otherembodiments, the light distributing elements 730, 740, 750 may bedisposed on a bottom major surface of associated waveguides 670, 680,690, respectively. In some other embodiments, the light distributingelements 730, 740, 750 may be disposed on both top and bottom majorsurface of associated waveguides 670, 680, 690 respectively; or thelight distributing elements 730, 740, 750, may be disposed on differentones of the top and bottom major surfaces in different associatedwaveguides 670, 680, 690, respectively.

The waveguides 670, 680, 690 may be spaced apart and separated by, e.g.,gas, liquid, or solid layers of material. For example, as illustrated,layer 760 a may separate waveguides 670 and 680; and layer 760 b mayseparate waveguides 680 and 690. In some embodiments, the layers 760 aand 760 b are formed of low refractive index materials (that is,materials having a lower refractive index than the material forming theimmediately adjacent one of waveguides 670, 680, 690). Preferably, therefractive index of the material forming the layers 760 a, 760 b is 0.05or more, or 0.10 or less than the refractive index of the materialforming the waveguides 670, 680, 690. Advantageously, the lowerrefractive index layers 760 a, 760 b may function as cladding layersthat facilitate TIR of light through the waveguides 670, 680, 690 (e.g.,TIR between the top and bottom major surfaces of each waveguide). Insome embodiments, the layers 760 a, 760 b are formed of air. While notillustrated, it will be appreciated that the top and bottom of theillustrated set 660 of waveguides may include immediately neighboringcladding layers.

Preferably, for ease of manufacturing and other considerations, thematerial forming the waveguides 670, 680, 690 are similar or the same,and the material forming the layers 760 a, 760 b are similar or thesame. In some embodiments, the material forming the waveguides 670, 680,690 may be different between one or more waveguides, or the materialforming the layers 760 a, 760 b may be different, while still holding tothe various refractive index relationships noted above.

With continued reference to FIG. 9A, light rays 770, 780, 790 areincident on the set 660 of waveguides. It will be appreciated that thelight rays 770, 780, 790 may be injected into the waveguides 670, 680,690 by one or more image injection devices 360, 370, 380, 390, 400 (FIG.6).

In some embodiments, the light rays 770, 780, 790 have differentproperties, e.g., different wavelengths or different ranges ofwavelengths, which may correspond to different colors. The in-couplingoptical elements 700, 710, 720 each deflect the incident light such thatthe light propagates through a respective one of the waveguides 670,680, 690 by TIR.

For example, in-coupling optical element 700 may be configured todeflect ray 770, which has a first wavelength or range of wavelengths.Similarly, the transmitted ray 780 impinges on and is deflected by thein-coupling optical element 710, which is configured to deflect light ofa second wavelength or range of wavelengths. Likewise, the ray 790 isdeflected by the in-coupling optical element 720, which is configured toselectively deflect light of third wavelength or range of wavelengths.

With continued reference to FIG. 9A, the deflected light rays 770, 780,790 are deflected so that they propagate through a correspondingwaveguide 670, 680, 690; that is, the in-coupling optical elements 700,710, 720 of each waveguide deflects light into that correspondingwaveguide 670, 680, 690 to in-couple light into that correspondingwaveguide. The light rays 770, 780, 790 are deflected at angles thatcause the light to propagate through the respective waveguide 670, 680,690 by TIR. The light rays 770, 780, 790 propagate through therespective waveguide 670, 680, 690 by TIR until impinging on thewaveguide's corresponding light distributing elements 730, 740, 750.

With reference now to FIG. 9B, a perspective view of an example of theplurality of stacked waveguides of FIG. 9A is illustrated. As notedabove, the in-coupled light rays 770, 780, 790, are deflected by thein-coupling optical elements 700, 710, 720, respectively, and thenpropagate by TIR within the waveguides 670, 680, 690, respectively. Thelight rays 770, 780, 790 then impinge on the light distributing elements730, 740, 750, respectively. The light distributing elements 730, 740,750 deflect the light rays 770, 780, 790 so that they propagate towardsthe out-coupling optical elements 800, 810, and 820, respectively.

In some embodiments, the light distributing elements 730, 740, 750 areorthogonal pupil expanders (OPE's). In some embodiments, the OPE's bothdeflect or distribute light to the out-coupling optical elements 800,810, 820 and also increase the beam or spot size of this light as itpropagates to the out-coupling optical elements. In some embodiments,e.g., where the beam size is already of a desired size, the lightdistributing elements 730, 740, 750 may be omitted and the in-couplingoptical elements 700, 710, 720 may be configured to deflect lightdirectly to the out-coupling optical elements 800, 810, 820. Forexample, with reference to FIG. 9A, the light distributing elements 730,740, 750 may be replaced with out-coupling optical elements 800, 810,820, respectively. In some embodiments, the out-coupling opticalelements 800, 810, 820 are exit pupils (EP's) or exit pupil expanders(EPE's) that direct light in a viewer's eye 210 (FIG. 7). It will beappreciated that the OPE's may be configured to increase the dimensionsof the eye box in at least one axis and the EPE's may be to increase theeye box in an axis crossing, e.g., orthogonal to, the axis of the OPEs.

Accordingly, with reference to FIGS. 9A and 9B, in some embodiments, theset 660 of waveguides includes waveguides 670, 680, 690; in-couplingoptical elements 700, 710, 720; light distributing elements (e.g.,OPE's) 730, 740, 750; and out-coupling optical elements (e.g., EP's)800, 810, 820 for each component color. The waveguides 670, 680, 690 maybe stacked with an air gap/cladding layer between each one. Thein-coupling optical elements 700, 710, 720 redirect or deflect incidentlight (with different in-coupling optical elements receiving light ofdifferent wavelengths) into its waveguide. The light then propagates atan angle that will result in TIR within the respective waveguide 670,680, 690. In the example shown, light ray 770 (e.g., blue light) isdeflected by the first in-coupling optical element 700, and thencontinues to bounce down the waveguide, interacting with the lightdistributing element (e.g., OPE's) 730 and then the out-coupling opticalelement (e.g., EPs) 800, in a manner described earlier. The light rays780 and 790 (e.g., green and red light, respectively) will pass throughthe waveguide 670, with light ray 780 impinging on and being deflectedby in-coupling optical element 710. The light ray 780 then bounces downthe waveguide 680 via TIR, proceeding on to its light distributingelement (e.g., OPEs) 740 and then the out-coupling optical element(e.g., EP's) 810. Finally, light ray 790 (e.g., red light) passesthrough the waveguide 690 to impinge on the light in-coupling opticalelements 720 of the waveguide 690. The light in-coupling opticalelements 720 deflect the light ray 790 such that the light raypropagates to light distributing element (e.g., OPEs) 750 by TIR, andthen to the out-coupling optical element (e.g., EPs) 820 by TIR. Theout-coupling optical element 820 then finally out-couples the light ray790 to the viewer, who also receives the out-coupled light from theother waveguides 670, 680.

FIG. 9C illustrates a top-down plan view of an example of the pluralityof stacked waveguides of FIGS. 9A and 9B. As illustrated, the waveguides670, 680, 690, along with each waveguide's associated light distributingelement 730, 740, 750 and associated out-coupling optical element 800,810, 820, may be vertically aligned. However, as discussed herein, thein-coupling optical elements 700, 710, 720 are not vertically aligned;rather, the in-coupling optical elements are preferably non-overlapping(e.g., laterally spaced apart as seen in the top-down view). Asdiscussed further herein, this non-overlapping spatial arrangementfacilitates the injection of light from different resources intodifferent waveguides on a one-to-one basis, thereby allowing a specificlight source to be uniquely coupled to a specific waveguide. In someembodiments, arrangements including non-overlapping spatially separatedin-coupling optical elements may be referred to as a shifted pupilsystem, and the in-coupling optical elements within these arrangementsmay correspond to sub pupils.

Example Imaging Systems for Off-Axis Imaging

As described above, the eyes or tissue around the eyes of the wearer ofa HMD (e.g., the wearable display system 200 shown in FIG. 2) can beimaged using multiple coupling optical elements to direct light from theeye through a substrate and into a camera assembly. The resulting imagescan be used to track an eye or eyes, image the retina, reconstruct theeye shape in three dimensions, extract biometric information from theeye (e.g., iris identification), etc.

As outlined above, there are a variety of reasons why a HMD might useinformation about the state of the eyes of the wearer. For example, thisinformation can be used for estimating the gaze direction of the weareror for biometric identification. This problem is challenging, however,because of the short distance between the HMD and the wearer's eyes. Itis further complicated by the fact that gaze tracking requires a largerfield of view, while biometric identification requires a relatively highnumber of pixels on target on the iris. For an imaging system that willattempt to accomplish both of these objectives, the requirements of thetwo tasks are largely at odds. Finally, both problems are furthercomplicated by occlusion by the eyelids and eyelashes. Embodiments ofthe imaging systems described herein may address at least some of theseproblems.

FIGS. 10A and 10B schematically illustrate an example of an imagingsystem 1000 a configured to image one or both eyes 210, 220 of a wearer90. The imaging system 1000 a comprises a substrate 1070 and a cameraassembly 1030 arranged to view the eye 220. Embodiments of the imagingsystem 1000 a described herein with reference to FIGS. 10A and 10B canbe used with HMDs including the display devices described herein (e.g.,the wearable display system 200 shown in FIG. 2, the display system 250shown in FIGS. 6 and 7, and the stack 660 of FIGS. 9A-9C). For example,in some implementations where the imaging system 1000 a is part of thedisplay system 250 of FIG. 6, the substrate 1070 may replace one of thewaveguides 270, 280, 290, 300, or 310, may be disposed between the ofwaveguide stack 260 and eye 210, or may be disposed between thewaveguide stack 260 and the world 510.

In some embodiments, the camera assembly 1030 may be mounted inproximity to the wearer's eye, for example, on a frame 80 of thewearable display system 60 of FIG. 2 (e.g., on an ear stem 82 near thewearer's temple); around the edges of the display 70 of FIG. 2 (as shownin FIG. 10B); or embedded in the display 70 of FIG. 2. The cameraassembly 1030 may be substantially similar to camera assembly 630 ofFIG. 6. In other embodiments, a second camera assembly can be used forseparately imaging the wearer's other eye 210. The camera assembly 1030can include an IR digital camera that is sensitive to IR radiation. Thecamera assembly 1030 can be mounted so that it is forward facing (e.g.,in the direction of the wearer's vision toward), as illustrated in FIG.10A, or the camera assembly 1030 can be mounted to be facing backwardand directed at the eye 220 (e.g., FIG. 10B).

In some embodiments, the camera assembly 1030 may include an imagecapture device and a light source 1032 to project light to the eye 220,which may then be reflected by the eye 220 and detected by the cameraassembly 1030. While the light source 1032 is illustrated as attached tothe camera assembly 1030, the light source 1032 may be disposed in otherareas with respect to the camera assembly such that light emitted by thelight source is directed to the eye of the wearer and reflected to thecamera assembly 1030. For example, where the imaging system 1000 a ispart of the display system 250 (FIG. 6) and the substrate 1070 replacesone of waveguides 270, 280, 290, 300, or 310, the light source 1032 maybe one of light emitters 360, 370, 380, 390, or light source 530.

In the embodiment illustrated in FIG. 10A, the camera assembly 1030 ispositioned to view a proximal surface 1074 of the substrate 1070. Thesubstrate 1070 can be, for example, a portion of the display 70 of FIG.2 or a lens in a pair of eyeglasses. The substrate 1070 can betransmissive to at least 10%, 20%, 30%, 40%, 50%, or more of visiblelight incident on the substrate 1070. In other embodiments, thesubstrate 1070 need not be transparent (e.g., in a virtual realitydisplay). The substrate 1070 can comprise one or more coupling opticalelements 1078. In some embodiments, the coupling optical elements 1078may be selected to reflect a first range of wavelengths while beingsubstantially transmissive to a second range of wavelengths differentfrom the first range of wavelengths. In some embodiments, the firstrange of wavelengths can be IR wavelengths, and the second range ofwavelengths can be visible wavelengths. The substrate 1070 may comprisea polymer or plastic material such as polycarbonate or other lightweightmaterials having the desired optical properties. Without subscribing toa particular scientific theory, plastic materials may be less rigid andthus less susceptible to breakage or defects during use. Plasticmaterials may also be lightweight, thus, when combined with the rigidityof the plastic materials allowing thinner substrates, may facilitatemanufacturing of compact and light weight imaging systems. While thesubstrate 1070 is described as comprising a polymer such aspolycarbonate or other plastic having the desired optical properties,other materials are possible, such as glass having the desired opticalproperties, for example, fused silica.

The coupling optical elements 1078 can comprise a reflective opticalelement configured to reflect or redirect light of a first range ofwavelengths (e.g., IR light) while transmitting light of a second rangeof wavelengths (e.g., visible light). In such embodiments, IR light 1010a, 1012 a, and 1014 a from the eye 220 propagates to and reflects fromthe coupling optical elements 1078, resulting in reflected IR light 1010b, 1012 b, 1014 b which can be imaged by the camera assembly 1030. Insome embodiments, the camera assembly 1030 can be sensitive to or ableto capture at least a subset (such as a non-empty subset or a subset ofless than all) of the first range of wavelengths reflected by thecoupling optical elements 1078. For example, where the coupling opticalelements 1078 is a reflective element, the coupling optical elements1078 may reflect IR light in the a range of 700 nm to 1.5 μm, and thecamera assembly 1030 may be sensitive to or able to capture near IRlight at wavelengths from 700 nm to 900 nm. As another example, thecoupling optical elements 1078 may reflect IR light in the a range of700 nm to 1.5 μm, and the camera assembly 1030 may include a filter thatfilters out IR light in the range of 900 nm to 1.5 μm such that thecamera assembly 1030 can capture near IR light at wavelengths from 700nm to 900 nm.

Visible light from the outside world (e.g., world 510 of FIG. 6) can betransmitted through the substrate 1070 and perceived by the wearer. Ineffect, the imaging system 1000 a can act as if there were a virtualcamera assembly 1030 c directed back toward the wearer's eye 220capturing a direct view image of the eye 220. Virtual camera assembly1030 c is labeled with reference to “c” because it may image virtual IRlight 1010 c, 1012 c, and 1014 c (shown as dotted lines) propagated fromthe wearer's eye 220 through the substrate 1070. Although couplingoptical elements 1078 is illustrated as disposed on the proximal surface1074 of the substrate 1070, other configurations are possible. Forexample, the coupling optical elements 1078 can be disposed on a distalsurface 1076 of the substrate 1060 or within the substrate 1070. Inimplementations where the substrate 1070 is part of display system 250of FIG. 6, the coupling optical element 1078 may be an out-couplingoptical element 570, 580, 590, 600, or 610.

While an example arrangement of imaging system 1000 a is shown in FIG.10A, other arrangements are possible. For example, multiple couplingoptical elements may be used and configured to in-couple light into thesubstrate 1070 via TIR and out-couple the light to the camera assembly1030, for example, as will be described in connection to FIGS. 11-18.While the coupling optical elements 1078 have been described asreflective optical elements, other configurations are possible. Forexample, the coupling optical elements 1078 may be a transmissivecoupling optical element that substantially transmits a first and asecond range of wavelengths. The transmissive coupling optical elementmay refract a first wavelength at an angle, for example, to induce TIRwithin the substrate 1070, while permitting the second range ofwavelengths to pass substantially unhindered.

Example Imaging Systems for Off-Axis Imaging Using Multiple CouplingOptical Elements

FIG. 11 schematically illustrates another example imaging system 1000 bcomprising multiple coupling optical elements to totally internallyreflect light from an object through a substrate 1070 to image an objectat a camera assembly 1030. FIG. 11 illustrates an embodiment of imagingsystem 1000 b comprising a substrate 1070 comprising at least twocoupling optical elements 1178 a, 1188 a disposed on one or moresurfaces of the substrate 1070 and a camera assembly 1030 arranged toview an object positioned at an object plane 1120. While a specificarrangement is depicted in FIG. 11, this is for illustrative purposesonly and not intended to be limiting. Other optical elements (forexample, lenses, waveguides, polarizers, prisms, etc.) may be used tomanipulate the light from the object so to focus, correct aberrations,direct, etc., the light as desired for the specific application.

In the embodiment of FIG. 11, the substrate 1070 includes two couplingoptical elements 1178 a, 1188 a, each disposed adjacent to the distaland proximal surfaces 1076, 1074 of the substrate 1070, respectively. Insome embodiments, the coupling optical elements 1178 a, 1188 a may beattached or fixed to the surfaces of the substrate 1070. In otherembodiments, one or more of the coupling optical element 1178 a, 1188 amay be embedded in the substrate 1070 or etched onto the surfaces of thesubstrate 1070. Yet, in other embodiments, alone or in combination, thesubstrate 1070 may be manufactured to have a region comprising thecoupling optical elements 1178 a, 1188 a as part of the substrate 1070itself. While an example arrangement of the coupling optical elements1178 a, 1188 a is shown in FIG. 11, other configurations are possible.For example, coupling optical elements 1178 a, 1188 a may both bepositioned adjacent to the distal surface 1076 or proximal surface 1074(as illustrated in FIGS. 12A, 13A, 13B, and 14B) or coupling opticalelements 1178 a may be positioned on the proximal surface 1074 whilecoupling optical elements 1188 a is positioned on the distal surface1076 (as illustrated in FIG. 14A).

The coupling optical elements 1178 a and 1188 a may be similar to thecoupling optical elements 1078 of FIGS. 10A and 10B. For example, FIG.11 illustrates the imaging system 1000 b where both coupling opticalelements 1178 a, 1188 a are reflective coupling optical elements thatare wavelength selective, such that they selectively redirect one ormore wavelengths of light, while transmitting other wavelengths oflight, as described above in connection to FIG. 10A. In someembodiments, the coupling optical elements 1178 a and 1188 a deflectlight of a first wavelength range (e.g., IR light, near-IR light, etc.)while transmitting a second wavelength range (e.g., visible light). Asdescribed below, the coupling optical elements 1178 a, 1188 a maycomprise diffractive features forming a diffraction patter (e.g., aDOE).

Referring to FIG. 11, the camera assembly 1030 is mounted backwardfacing toward the object plane 1120 and viewing the distal surface 1076.In various embodiments, the camera assembly 1030 may be mounted inproximity to the wearer's eye (for example on the frame 80 of FIG. 2)and may include light source 1032 (not shown in FIG. 11). The cameraassembly 1030 can include an IR digital camera that is sensitive to IRradiation. While the camera assembly 1030 of FIG. 11 is shown asbackward facing, other arrangements are possible. For example, cameraassembly 1030 can be mounted so that it is forward facing.

In some embodiments, an object (e.g., the eye 220 or a part thereof) atthe object plane 1120 may be illuminated by the light source 1032 (FIGS.10A and 10B). For example, where the pupil is to be imaged, the lightsource 1032 is directed thereto and illuminates the pupil of eye 220. Inother embodiments, the first Purkinje image, which is the virtual imageformed by the reflection of a point source off the anterior surface ofthe cornea may be imaged. Any physical or optical object associated withthe eye that can be uniquely identified and that will indicate eyeposition, pupil position, or gaze direction may be imaged. Uponillumination, the object may reflect the light toward the substrate 1070as light rays 1122 a-e (collectively referred to hereinafter as “1122”).For example, light rays 1122 a-e may be illustrative of diffuse lightreflected from the pupil, iris, eyelid, sclera, other tissue around theeye, etc. In another example, light rays 1122 a-e may be illustrative ofspecularly reflected light from a glint (e.g., a Purkinje image).Without subscribing to a scientific theory, a reflection from the eye,parts of the eye, or tissue around the eye may rotate the polarizationof the incident light depending on the orientation of the illumination.In some embodiments, the light source 1032 (FIGS. 10A and 10B) may be aLED light source that does not have a specific polarization, unless apolarizer is implemented in the optical path with may reduce theintensity of the light, for example, by as much of 50%. While only lightrays 1122 are shown in FIG. 11, this is for illustrative purposes onlyand any number of reflected light rays are possible. Each of light rays1122 may be reflected at the same or different angles from the object.For example, FIG. 11 illustrates that light ray 1122 a is reflected at afirst angle that may be larger than the angle at which light ray 1122 eis reflected from the object. Other configurations are possible.

While the above description referred to light rays 1122 as reflectedfrom the object, other configurations are possible. In some embodiments,the light rays 1122 are emitted by a light source located at the objectplane 1120 instead of reflecting light from the source 1032 (FIGS. 10Aand 10B). As such, the light rays 1122 may be directed toward thesubstrate 1070. It will be understood that light rays 1122 may be all orsome of the light reflected from or emitted by the object plane 1120.

As illustrated in FIG. 11, upon emanating from the object plane 1120,the light rays 1122 are incident on the proximal surface 1074 of thesubstrate at an angle of incidence relative to an imaginary axisperpendicular to the proximal surface 1074 at the point of incidence.The light rays 1122 then enter the substrate 1070 and are refractedbased, in part, on angle of incidence at the proximal surface 1074 andthe ratio of the refractive indices of the substrate 1070 and the mediumimmediately adjacent to the proximal surface 1074.

The light rays 1122 travel to and impinge upon the coupling opticalelement 1178 a at an angle of incidence relative to an imaginary axisperpendicular to the distal surface 1076 at the point of incidence. Thelight rays 1122 are deflected by the coupling optical element 1178 a sothat they propagate through the substrate 1070; that is, the couplingoptical element 1178 a functions as a reflective in-coupling opticalelement that reflects the light into the substrate 1070. The light rays1122 are reflected at angles such that the in-coupled light rays 1122propagate through the substrate in lateral direction toward the couplingoptical element 1188 a by total internal reflection. Without subscribingto any scientific theory, the total internal reflection condition can besatisfied when the diffraction angle θ between the incident light andthe perpendicular axis is greater than the critical angle, θ_(C), of thesubstrate 1070. Under some circumstances, the total internal reflectioncondition can be expressed as:

sin(θ_(C))=n _(o)/n_(s)   [1]

where n_(s) is the refractive index of the substrate 1070 and n_(o) isthe refractive index of the medium adjacent to the surface substrate1070. According to various embodiments, n_(s) may be between about 1 andabout 2, between about 1.4 and about 1.8, between about 1.5 and about1.7, or other suitable range. For example, the substrate 1070 maycomprise a polymer such as polycarbonate or a glass (e.g., fused silica,etc.). In some embodiments, the substrate 1070 may be 1 to 2 millimetersthick, from the proximal surface 1074 to the distal surface 1076. Forexample, the substrate 1070 may be a 2 millimeter thick portion of fusedsilica or a 1 millimeter thick portion of polycarbonate. Otherconfigurations are possible to achieve the desired operation and imagequality at the camera assembly 1030.

In some embodiments, the substrate 1070 may be formed of high refractiveindex material (e.g., materials having a higher refractive index thanthe medium immediately adjacent to the substrate 1070). For example, therefractive index of the material immediately adjacent to the substrate1070 may be less than the substrate refractive index by 0.05 or more, or0.10 or more. Without subscribing to a particular scientific theory, thelower refractive index medium may function to facilitate TIR of lightthrough the substrate 1070 (e.g., TIR between the proximal and distalsurfaces 1074, 1076 of the substrate 1070). In some embodiments, theimmediately adjacent medium comprises air with a refractive index no ofabout 1. Critical angles can be in a range from 20 degrees to 50degrees, depending on the substrate material and surrounding medium. Inother embodiments, alone or in combination, the immediately adjacentmedium may comprise other structures and layers, for example, one ormore of the layers described in connection to FIGS. 6 and 9A-9C may beimmediately adjacent to either the proximal or distal surface 1074, 1076of the substrate 1070.

The light then propagates through the substrate 1070 in a directiongenerally parallel with the surfaces of the substrate 1070 and towardthe coupling optical element 1188 a. Generally toward may refer to thecondition that the light rays 1122 are reflected between the surfaces ofthe substrate 1070 and as such travel in directions that may not beexactly parallel to the substrate 1070, but the overall direction oftravel is substantially parallel with the surfaces of the substrate. Thelight rays 1122 propagate through the substrate 1070 by TIR untilimpinging on the coupling optical element 1188 a. Upon reaching thecoupling optical element 1188 a, the light rays 1122 are deflected sothat they propagate out of the substrate 1070; that is, the couplingoptical element 1188 a functions as a reflective out-coupling opticalelement that reflects the light out of the substrate 1070. The lightrays 1120 are reflected at angles such that the TIR condition is nolonger satisfied (e.g., the diffraction angle θ is less than thecritical angle θ_(c)). The coupling optical element 1188 a may alsoreflect the light rays 1122 at an angle toward the camera assembly 1030.For example, the light rays 1122 may be reflected at an angle so as toexit the substrate 1070, are refracted by the interface at the distalsurface 1076, and propagate to the camera assembly 1030. The cameraassembly 1030 then receives the light rays 1122 and images the objectplane 1120 based thereon.

While FIG. 11 illustrates a configuration in which light travels fromcoupling optical element 1178 a to coupling optical element 1188 a withtwo instances of total internal reflection, other configurations arepossible. For example, the light rays 1122 may be totally internallyreflected any number of times (e.g., 1, 2, 3, 4, 5, 6, 7, etc.) suchthat the light rays 1122 travel through the substrate 1070 toward thecamera assembly 1030. The camera assembly 1030 may thus be positionedanywhere and configured to capture a direct view image at some distancefrom the object. Without subscribing to a scientific theory, TIR maybeinclude highly efficiency, substantially lossless reflections, thus thenumber of times the light rays 1122 TIR may be selected based on thedesired position of the camera. However, in some embodiments, someleakage, even minimal, may occur at each reflection within the substrate1070. Thus, minimizing the number of reflections within the substrate1070 may reduce leakage of light and improve image capture performance.Furthermore, without subscribing to a scientific theory, reducing thenumber of reflections may improve image quality by reducing imageblurring or brightness reduction (e.g., fewer reflections may produce abrighter more intense image) due to impurity or non-uniform surfaces ofthe substrate 1070. Therefore, design of the imaging systems described,and the components thereof, may be optimized with these considerationsin mind so as to minimize the number of TIR events and position thecamera assembly 1030 as desired.

Efficient in- and out-coupling of light into the substrate 1070 can be achallenge in designing waveguide-based see-through displays, e.g., forvirtual/augmented/mixed reality display applications. For these andother applications, it may be desirable to include diffraction gratingsformed of a material whose structure is configurable to optimize variousoptical properties, including diffraction properties. The desirablediffraction properties may include, among other properties, polarizationselectivity, spectral selectivity, angular selectivity, high spectralbandwidth, and high diffraction efficiencies, among other properties. Toaddress these and other needs, in various embodiments disclosed herein,the coupling optical elements 1178 a, 1188 a may comprise diffractivefeatures that form a diffraction pattern, such as DOEs or diffractiongratings.

Generally, diffraction gratings have a periodic structure, which splitsand diffracts light into several beams traveling in differentdirections. The direction of the beams depends, among other things, onthe period of the periodic structure and the wavelength of the light.The period may be, in part, based on the grating spatial frequency ofthe diffractive features. To optimize certain optical properties, e.g.,diffraction efficiencies and reduce potential rainbow effects, forcertain applications such as in- and out-coupling light from thesubstrate 1070, various material properties of the DOE can be optimizedfor a given wavelength. For example, where IR light is used, the spatialfrequency of the DOEs 1178 a, 1188 a may between 600 and 2000 lines permillimeter. In one embodiment, the spatial frequency may beapproximately 1013 lines per millimeter (e.g., FIGS. 12A and 13A). Inone embodiment, the example DOE 1178 a of FIG. 11 may have 1013.95 linesper millimeter. In another embodiment, the spatial frequency isapproximately 1400 lines per millimeter, as described in connection toFIG. 15. Thus, the spatial frequency of the coupling optical elements1178 a, 1188 a may be, at least, one consideration when optimizing theimaging systems described herein. For example, the spatial frequency maybe selected to support TIR conditions. As another example, alone or incombination, the spatial frequency may be selected to maximize lightthroughput with minimum artifacts (e.g., ghost or duplicative images asdescribed in FIG. 12B) depending on the configuration and dimensions ofthe components of the imaging system. In some embodiments, thediffractive features may have any configurations; however the firstcoupling optical element 1178 a may be optimized to have minimal or novisual artifacts (e.g., rainbow effects) because the first couplingoptical element 1178 a may be positioned within the user's field ofview.

In some implementations, the DOE may be an off-axis DOE, an off-axisHolographic Optical Element (HOE), an off-axis holographic mirror(OAHM), or an off-axis volumetric diffractive optical element (OAVDOE).In some embodiments, an OAHM may have optical power as well, in whichcase it can be an off-axis volumetric diffractive optical element(OAVDOE). In some embodiments, one or more of the coupling opticalelements 1178 a, 1188 a may be an off-axis cholesteric liquid crystaldiffraction grating (OACLCG) which can be configured to optimize, amongother things, polarization selectivity, bandwidth, phase profile,spatial variation of diffraction properties, spectral selectivity andhigh diffraction efficiencies. For example, any of the CLCs or CLCGsdescribed in U.S. patent application Ser. No. 15/835,108, filed Dec. 7,2017, entitled “Diffractive Devices Based On Cholesteric LiquidCrystal,” which is incorporated by reference herein in its entirety forall it discloses, can be implemented as coupling optical elements asdescribed herein. In some embodiments, one or more coupling opticalelements 1178 a, 1188 a may be switchable DOEs that can be switchedbetween “on” states in which they actively diffract, and “off” states inwhich they do not significantly diffract.

In some embodiments, one or more of the coupling optical elements 1178a, 1188 a may be any reflective or transmissive liquid crystal gratings.The above described CLCs or CLCGs may be one example of a liquid crystalgrating. Other liquid crystal gratings may also include liquid crystalfeatures and/or patterns that have a size less than the wavelength ofvisible light and may comprise what are referred to asPancharatnam-Berry Phase Effect (PBPE) structures, metasurfaces, ormetamaterials. For example, any of the PBPE structures, metasurfaces, ormetamaterials described in U.S. Patent Publication No. 2017/0010466,entitled “Display System With Optical Elements For In-CouplingMultiplexed Light Streams”; U.S. patent application Ser. No. 15/879,005,filed Jan. 24, 2018, entitled “Antireflection Coatings ForMetasurfaces”; or U.S. patent application Ser. No. 15/841,037, filedDec. 13, 2017, entitled “Patterning Of Liquid Crystals UsingSoft-Imprint Replication Of Surface Alignment Patterns,” each of whichis incorporated by reference herein in its entirety for all itdiscloses, can be implemented as coupling optical elements as describedherein. Such structures may be configured for manipulating light, suchas for beam steering, wavefront shaping, separating wavelengths and/orpolarizations, and combining different wavelengths and/or polarizationscan include liquid crystal gratings with metasurface, otherwise referredto as metamaterials liquid crystal gratings or liquid crystal gratingswith PBPE structures. Liquid crystal gratings with PBPE structures cancombine the high diffraction efficiency and low sensitivity to angle ofincidence of liquid crystal gratings with the high wavelengthsensitivity of the PBPE structures.

In some embodiments, certain DOEs may provide non-limiting advantageswhen utilized as the coupling optical elements as described herein. Forexample, without subscribing to a scientific theory, liquid crystalgratings, CLCs, CLCGs, volume phase gratings, and meta-surface gratingsmay comprise optical properties configured to reduce or eliminate theappearance of visual artifacts, such as rainbow effects described aboveand herein. In some embodiments, when employing these DOEs, it may bedesirable to illuminate the DOE with polarized light (e.g., the lightrays 1122 may include a desired polarization) to maximize the throughputof light into the substrate 1070. However, as described above, the eyemay rotate the polarization of incident depending on the orientation,thus, in some embodiments, the light source 1030 may emit un-polarizedlight. The reflected light rays 1122 may also be un-polarized, thus aportion of the light may not be throughput due to the polarizationproperties of the DOE (e.g., up to 50% of the light ray 1122 may be lostat the coupling optical element 1178 a). In some embodiments, to improvethroughput, a double layer DOE may be used as the coupling opticalelement 1178 a. For example, a first DOE layer configured to operate atone polarization state and as second DOE layer configured to operate ata second polarization state.

For some embodiments, it may be desirable to use DOEs havingsufficiently high diffraction efficiency so that as much of the lightrays 1122 are in-coupled into the substrate 1070 and out-coupled towardthe camera assembly. Without subscribing to a scientific theory,relatively high diffraction efficiency may permit directingsubstantially all of the light received at the coupling optical element1178 a to the camera assembly 1030, thereby improving image quality andaccuracy. In some embodiments, the diffraction efficiency may be based,in part, on the sensitivity of the camera assembly 1030 (e.g., a highersensitivity may permit a lower diffraction efficiency). In variousembodiments, a DOE may be selected to have a high diffractive efficiencywith respect to a first range of wavelengths (e.g., IR light) and lowdiffractive efficiency in a second range of wavelengths (e.g., visiblelight). Without subscribing to a scientific theory, a low diffractiveefficiency with respect to visible light may reduce rainbow effects inthe viewing path of the user.

In some applications, a DOE may cause a rainbow effect when a user viewsvisible light through diffractive features. Without subscribing to aparticular scientific theory, the rainbow affect may be the result of arange of wavelengths interacting with the diffractive features, therebydeflecting different wavelengths (e.g., colors) in different directionsa different diffraction angles. In some embodiments described herein,the rainbow effect from the world interacting with the coupling opticalelements 1178 a, 1188 b as viewed by a user may be reduced by modifyingor controlling the diffractive features to reduce this effect. Forexample, since the diffraction angle of light on a DOE is based on theperiod or spatial frequency of the grating, the shape of the diffractivefeatures may be selected to concentrate the majority of the diffractedlight at a particular location for a given range of wavelengths (e.g., atriangular cross section or blazing).

In some embodiments, the substrate 1070 may be one of the waveguides270, 280, 290, 300, or 310 of FIG. 6. In this embodiment, thecorresponding out-coupling optical element 570, 580, 590, 600, or 610may be replaced with an in-coupling optical element 1178 a configured toinduce TIR of light reflected by the eye. In some embodiments, a portionof out-coupling optical element 570, 580, 590, 600, or 610 may bereplaced with an in-coupling optical element 1178, such that thecorresponding waveguide 270, 280, 290, 300, or 310 may be used asdescribed in connection to FIG. 6 and to direct light reflected tocamera assembly 630.

In some embodiments, the substrate 1070 may be one the waveguides 670,680, or 690 of FIGS. 9A-9C. In these embodiments, the correspondinglight distributing elements 800, 810, and 830, or a portion thereof, maybe replaced with the in-coupling optical element 1178 a, while thein-coupling optical element 700, 710, and 720, or portion thereof, maybe replaced with the out-coupling optical element 1188 a. In someembodiments, the OPEs 730, 740, and 750 may remain in the optical pathof the light traveling from the in-coupling optical element 1178 a tothe out-coupling optical element 1188 a. However, the OPEs 730, 740, and750 may be configured to distribute the light to out-coupling opticalelement 1188 a and also decrease the beam spot size as it propagates.

In various embodiments, the field of view of the camera assembly 1030 isconfigured to be sufficient to image the entire object plane 1120 (e.g.,the eye 220 of FIG. 10, a part thereof, or tissue surrounding the eye)throughout a variety of field positions. For example, in the exampleshown in FIG. 11 the size of the imaged object plane 1120 may be 30 mm(horizontal) by 16 mm (vertical). In some embodiments, the couplingoptical elements 1178 a, 1188 a are designed to be large enough to atleast match the size of the object to be imaged; that is the couplingoptical elements 1178 a, 1188 a are configured to receive light from thefull size of the imaged object. For example, the coupling opticalelement 1178 a receive light originating from the eye 220. The couplingoptical element 1188 may be sized so as to reflect substantially all ofthe light rays 1122 that propagate through the substrate 1070 toward thecamera assembly 1030.

In various embodiments, other optical elements may be positioned alongthe path the light rays 1122 travel. For example, intervening opticalelements may be included between the substrate 1070 and the object plane1120 for directing the light rays 1122 toward the substrate 1070 at thedesired angle. Intervening optical elements may be included between thecamera assembly 1030 and the substrate 1070 directing and focusing thelight rays 1122 toward the camera assembly 1030 so as to place thecamera assembly 1030 at any desired location. In some embodiments,intervening optical elements may be used to filter the light rays 1122,change polarization or correct for aberrations. For example, acorrective optical element may be positioned along the optical path ofthe light rays 1122 arranged to and configured to reduce or eliminateoptical aberrations introduced by the optical components of the imagingsystem or, where the imaging system is part of the display system 250 ofFIG. 6, other waveguides or optical elements.

Alternative Embodiments for Off-Axis Imaging Using Multiple CouplingOptical Elements

While FIG. 11 shows an example imaging system 1000 b comprising thesubstrate 1070 having coupling optical elements 1178 a, 1188 aconfigured to TIR light from the object plane 1120 through the substrate1070, other configurations are possible. For example, FIG. 11illustrates both coupling optical elements 1178 a, 1188 a as reflectivecoupling optical elements; however, one or both coupling opticalelements may be transmissive coupling optical elements configured torefract a first range of wavelengths at an angle satisfying the TIRconditions, while transmitting a second range of wavelengthssubstantially through the substrate 1070. FIGS. 12A-18 illustrate someembodiments of substrate 1070, however, other configurations arepossible.

FIG. 12A schematically illustrates an example imaging system 1000 c. Theimaging system 1000 c uses multiple coupling optical elements 1178 a,and 1188 b to TIR the light 1122 from an object plane 1220 through thesubstrate 1070. Similar to FIG. 11, FIG. 12A illustrates the couplingoptical element 1178 a as a reflective coupling optical element disposedon the distal surface 1076 of the substrate 1070 that in-couples thelight ray 1122 into the substrate 1070. However, while coupling opticalelement 1188 b is substantially similar to coupling optical element 1188a of FIG. 11, FIG. 12A illustrates a transmissive coupling opticalelement 1188 b disposed on the distal surface 1076 of the substrate1070. Thus, upon propagating through the substrate 1070 via TIR, thelight rays 1122 are reflected a third time on the proximal surface 1074toward the transmissive coupling optical element 1188 b. Thetransmissive coupling optical element 1188 b refracts the light rays1122 at an angle such that the TIR conditions no longer hold and thelight rays 1122 exit the substrate 1070. For example, where thetransmissive coupling optical element 1188 b is a DOE, the light isrefracted based on the spatial frequency of the DOE and aresubstantially deflected toward the camera assembly 1030.

FIG. 12A also illustrates a stray light ray 1222 that is captured by thecamera assembly 1030. For example, stray light ray 1222 is reflected bythe object 1120, but instead of propagating through the coupling opticalelements 1178 a, 1188 b, some or all of the stray light ray 1222 travelsdirectly toward the camera assembly 1030. Without subscribing to aparticular theory, the stray light ray 1222 is captured by the cameraassembly 1030, thereby producing a direct view image, as describedabove. Thus, the camera assembly 1030 may capture a direct view imagebased on the light ray 1222 (e.g., including the narrow FOV and defectsdescribed herein) along with the desired image based on the light rays1122 that TIR through the substrate. Since the camera assembly 1030captures light rays that have traveled along different optical paths,the final image would include various imperfections. One suchimperfection is illustrated in FIG. 12B, but others are possible.

FIG. 12B illustrates an example image 1210 of an object 1120 capturedusing the camera assembly 1030 of FIG. 12A. In the illustrative image1210, the camera assembly 1030 has captured an image 1210 of, forexample, a front face of a laser diode used as an object and illuminatedwith an IR light source. While a laser diode is illustrated in thisexample, other objects may be used to similar effect, for example an eye210 of a user. The image 1210 includes a direct view image 1205 of thelaser diode produced by light ray 1222 and set of images 1240 producedby light rays 1122. The set of images 1240 includes a desired off-axisimage (for illustrative purposes shown as image 1215) and multipleduplicative images (collectively illustrated as images 1212) fromdifferent perspectives. Such duplicative images 1212, in someembodiments, may require post-processing to synthesize a singleperspective image of the object if desired. In other embodiments, theimaging system may be designed to reduce or eliminate the un-wantedduplicative images 1212 and direct view image 1205 so as to capturesingle perspective image 1215.

For example, FIGS. 13A and B schematically illustrate another view ofthe imaging system 1000 c. FIGS. 13A and 13B illustrate example approachto reduce or eliminate the duplicative images 1212. Without subscribingto a particular scientific theory, the duplicative images 1212 may bereduced or substantially eliminated based on varying the thickness ofthe substrate 1070 (t), the size of the coupling optical elements 1178 a(d₁), and the stride distance (d₂) of the light rays 1122. The stridedistance (d₂) may refer to a distance parallel to the substrate 1070that a light ray travels as it reflects within the substrate; that is,for example, the distance between two adjacent points of incidence onthe distal surface 1076 of the substrate 1070 due to a single instanceof total internal reflection. In some embodiments, the direct view image1205 may also be reduced or removed, for example, by including a coatingon the proximal or distal surface 1074, 1076 close to the object 1220(e.g., an IR coating configured to block or reduce IR light from theobject 1220).

For example, ghost images can be reduce or eliminated by reducing thesize (d₁) of the coupling optical element 1178 a to the smallest sizeand varying the physical arrangement of the components of the imagingsystem 1000 c such that the stride distance (d₂) is greater than d₁.

In some embodiments, it may be desirable to control the stride distance(d₂) to achieve a large stride distance while minimizing the size of thecoupling optical element 1178 a. Without subscribing to a particularscientific theory, a large stride distance may reduce the intensity ofghost images or permit placement of the camera assembly 1030 outside ofthe stray light rays 1030. Thus, under some circumstances, the stridedistance can be expressed as:

d₂=2*t*tan(θ)   [2]

where θ is the diffraction angle of a light ray 1122 and t is thethickness of the substrate 1070. Increasing the stride distance may bedone by increasing the thickness (t) of the substrate or increasing thediffractive angle (θ). As described above, the diffractive angle (θ) maybe based on the spatial frequency or period of the diffractive features.For example, the lowest light ray 1122 e has the smallest diffractiveangle (θ), thus to increase the stride distance it may be preferable toincrease this diffractive angle. Furthermore, increasing the thicknessof the substrate 1070 may also increase the stride distance. However, itmay be desirable to balance the thickness of the substrate 1070 againstproducing lightweight and compact imaging systems. In one embodiment,the substrate 1070 is a 2.5 millimeter thick piece of polycarbonate(other materials are possible) and the grating spatial frequency is 720lines per millimeter. Various embodiments may include differentsubstrate thicknesses or grating spatial frequencies.

FIGS. 14A and 14B schematically illustrate the examples of imagingsystems with multiple coupling optical elements having an arrangementthat is different than the imaging system 1000 a of FIG. 11. Asdescribed in FIG. 11, the coupling optical elements are configured aseither in- or out-coupling optical elements for inducing TIR anddirecting the light rays 1122 through the substrate 1070 to the cameraassembly 1030. FIGS. 14A and 14B differ in the variation of the type andplacement of the coupling optical elements.

For example, FIG. 14A depicts the imaging system 1000 d that issubstantially similar to the imaging system 1000 b of FIG. 11. However,the imaging system 1000 d comprises a transmissive coupling opticalelement 1178 b disposed on the proximal surface 1074 of the substrate1070 and a transmissive coupling optical element 1188 b disposed on thedistal surface 1076 of the substrate 1070. The transmissive couplingoptical element 1178 b may be configured as an in-coupling opticalelement that is transmissive to but diffracts the light 1122 of FIG. 11at a diffraction angle to induce TIR at the distal surface 1046. Thelight 1122 may then be directed toward the transmissive coupling opticalelement 1188 b configured as an out-coupling optical element, asdescribed above in connection to FIG. 12A.

In the embodiment of FIG. 14B, the imaging system 1000 e issubstantially similar to the imaging system 1000 b of FIG. 11. However,the imaging system 1000 e comprises a transmissive coupling opticalelement 1178 b and a reflective coupling optical element 1188 a disposedon the proximal surface 1074 of the substrate 1070. The transmissivecoupling optical element 1178 b may be configured as an in-couplingoptical element transmissive to but diffracts the light 1122 of FIG. 11at a diffraction angle to induce TIR at the distal surface 1046. Thelight 1122 may then be directed toward the reflective coupling opticalelement 1188 a configured as an out-coupling optical element, asdescribed above in connection to FIG. 11.

FIG. 15 schematically illustrates another example imaging system 1000 fthat is substantially similar to imaging system 1000 c of FIGS. 12A-13B.Similar to the above imaging systems, FIG. 15 illustrates the imagingsystem 1000 f comprising the reflective coupling optical element 1178 aand the transmissive coupling optical element 1188 b disposed on thedistal surface 1076 of the substrate 1070. However, the coupling opticalelements 1178 a and 1188 b comprise a spatial frequency of 1411.765lines per millimeter and a pitch of 708.33 nanometers and the substrateis a 1 millimeter thick piece of polycarbonate. Accordingly, relative tothe imaging system 1000 c of FIGS. 12A-13B, the light 1122 may TIRmultiple times within the substrate 1070 and the camera assembly may beshifted further away from the coupling optical element 1178 a. Otherconfigurations are possible.

Alternative Embodiments of Imaging Systems for Off-Axis Imaging

While FIG. 11 shows an example imaging system 1000 b comprising thesubstrate 1070 having coupling optical elements 1178 a, 1188 aconfigured to TIR light from the object plane 1120 through the substrate1070, other configurations are possible.

For example, FIG. 16 illustrates an imaging system 1000 g comprising asubstrate 1070 disposed adjacent to an optical component 1650. In someembodiments, the optical component 1650 may be the waveguide stack 260of FIG. 6 or the waveguide stack 660 of FIGS. 9A-9C. While the substrate1070 is illustrated as adjacent to and between the object 1120 and theoptical component 1650, other configurations are possible. For example,the optical component 1650 may be between the substrate 1070 and theobject 1120 or the substrate 1070 may be part of the optical component1650. The substrate 1070 may comprise multiple reflective elements 1678and 1688. As illustrated in FIG. 16, the light 1122 may travel from theobject 1120 toward the substrate 1070 and interact with the proximalsurface 1074. The light 1122 may be refracted and directed to reflectiveelement 1678, which reflects the light 1122 at an angle such that thelight TIRs on the proximal surface 1074. Thus, the light 1122 travelstoward the reflective element 1688 via TIR. The light 1122 may bereflected by the reflective element 1688 toward the camera assembly1030. Accordingly, the camera assembly 1030 may capture an off-axisimage of the object 1120, as if the camera assembly 1030 was directlyviewing the object 1120 (e.g., virtual camera assembly 1030 c). In someembodiments, one or more of the reflective elements 1678, 1688 may be“hot mirrors” or comprise reflective coatings that are reflective in theIR while being transmissive in the visible spectrum.

In one embodiment of FIG. 16, the substrate 1070 is a 2 millimeter thickpiece of polycarbonate and the proximal surface 1074 is positioned 15.7millimeters to the right of the object plane 1120 (e.g., z-direction).The object plane 1120 is 12 millimeters vertically (e.g., y-direction).In some embodiments, the reflective element 1678 is configured tocapture a substantially full FOV, where the central light ray 1122 cpropagates at 25 degrees down (e.g., negative y-direction) from normal.The camera assembly 1030 may be positioned 15.7 millimeters down fromthe origination of the light ray 1122 c and 18.79 millimeters to theright. In this arrangement, the imaging system 1000 g captures an imageas if view from the virtual camera 1030 c positioned 10.56 millimetersdown and 22.65 millimeters to the right.

FIG. 17 illustrates an imaging system 1000 h comprising a substrate 1770disposed adjacent to an optical component 1650 (e.g., an opticalcover-glass or a prescription glass), and a reflective surface 1778disposed adjacent to the substrate 1770. In some embodiments, thesubstrate 1770 may be substantially similar to the substrate 1070described above. While a specific arrangement is shown in FIG. 17, otherconfigurations are possible. For example, the optical component 1650 maybe between the substrate 1650 and the object 1120 or the substrate 1770may be part of the optical component 1650. As illustrated in FIG. 17,the light 1122 may travel from the object 1120 toward the opticalcomponent 1650 and interact therewith. The light 1122 may then berefracted or pass through the optical component 1650 as it travelstoward the substrate 1770. After passing through the substrate 1770(refracted or passed through), the light 1122 is incident upon thereflective surface 1778. The reflective surface 1778 may have opticalproperties configured to reflect and direct the light 1122 toward thecamera assembly 1030. Accordingly, the camera assembly 1030 may capturean off-axis image of the object 1120, as if the camera assembly 1030 wasdirectly viewing the object 1120. In one embodiment of FIG. 17, theimaging system 1000 f is configured to capture an object plane 1120 thatis 16 millimeters by 24 millimeters, where the central light ray 1122 cpropagates at positive 17 degrees from normal (shown as line 1790).

In some embodiments the reflective surface 1778 may be a surface of adecorative or cosmetic lens or optical component. For example, adecorative lens may be a lens for use as sunglasses to filter outsunlight. In another embodiment, the decorative lens may be a colorfiltering lens for use in goggles. In yet other embodiments, thedecorative lens may have a colored visual appearance that is viewable byother people who are not wearing the lens (e.g., a lens that appearsblue, red, etc. to other people). The decorative lens may also include acolor layer that is viewed by people other than the user. The reflectivesurface 1778 may be a reflective coating on the inside surface of thedecorative lens. The reflective coating may be reflective in the IRwhile being transmissive in the visible spectrum so that the wearer isable to view the world. As shown in FIG. 17, the reflective surface 1778may comprise a concave shape configured to direct the light 1122 towardthe camera assembly 1030. However, other configurations are possible.

FIG. 18 illustrates an imaging system 1000 i comprising a substrate 1770disposed adjacent to an optical component 1850 and a prism 1878 disposedadjacent to the substrate 1770. In some embodiments, the substrate 1770may be substantially similar to the substrate 1070 described above. Theoptical component 1850 may be substantially similar to optical component1650, but may also comprise one or more of the exit pupil expanders 800,810, 820 of FIGS. 9A-9C. While a specific arrangement is shown in FIG.18, other configurations are possible. For example, the opticalcomponent 1850 may be between the substrate 1770 and the object 1120 orthe substrate 1770 may be part of the optical component 1850. Asillustrated in FIG. 18, the light 1122 may travel from the object 1120toward the optical component 1850 and interact therewith. The light 1122may be refracted or passed through as it travels toward the opticalcomponent 1850. After passing through the optical component 1850(refracted or passed through), the light 1122 enters prism 1878 and isreflected by surface 1878 a toward the camera assembly 1030.Accordingly, the camera assembly 1030 may capture an off-axis image ofthe object 1120, as if the camera assembly 1030 was directly viewing theobject 1120. In some embodiments, the prism may be an IR prism, “hotmirror,” or the surface 1878 a may comprise reflective coatings that arereflective in the IR while being transmissive in the visible spectrum.In one embodiment of FIG. 18, the imaging system 1000 i comprises acamera assembly 1030 having a vertical FOV of 35 degrees and a focaldistance of 30.73 millimeters. Such an imaging system 1000 i may beconfigured to capture an object plane 1120 that is 16 millimeters by 24millimeters, where the central light ray 1122 c propagates at negative25 degrees from normal (shown as line 1790).

Example Routine for Imaging an Object

FIG. 19 is a process flow diagram of an illustrative routine for imagingan object (e.g., an eye of the user) using an off-axis camera (e.g.,camera assembly 630 of FIG. 6 or camera assembly 1030 of FIG. 10A). Theroutine 1900 describes how a light from an object can be can be directedto a camera assembly that is positioned away from or off-axis relativeto the object for imaging the object as though the camera assembly waspointed directly toward the object.

At block 1910, an imaging system is provided that is configured toreceive light from the object and direct the light to a camera assembly.The imaging system may be one or more of the imaging systems 1000 a-i asdescribed above in connection to FIGS. 10A-11, 12A, and 13A-18. Forexample, the imaging system may comprise a substrate (e.g., substrate1070) comprising a first coupling optical element (e.g., first couplingoptical element 1078, 1178 a, or 1178 b) and a second coupling opticalelement (e.g., second optical element 1188 a or 1188 b). The first andsecond optical elements may be disposed on a distal surface or aproximal surface of the substrate as described above and throughout thisdisclosure. The first and second optical elements may be laterallyoffset from each other along the substrate 1070. As described above andthroughout this disclosure, the first coupling optical element may beconfigured to deflect light at an angle to TIR the light between theproximal and distal surfaces. The first optical element may beconfigured to deflect light at an angle generally toward the secondcoupling optical element. The second coupling optical element may beconfigured to receive the light from the first coupling optical elementand deflect the light at an angle out of the substrate.

At block 1920, the light is captured with a camera assembly (e.g.,camera assembly 630 of FIG. 6 or camera assembly 1030 of FIGS. 10A-11,12A, and 13A-18). The camera assembly may be orientated toward thesecond coupling optical element and to receive the light deflected bythe second coupling optical element. The camera assembly may be anoff-axis camera in a forward facing or backward facing configuration. Atblock 1930, an off-axis image of the object may be produced based on thecaptured light, as described herein and throughout this disclosure.

In some embodiments, the routine 1900 may include an optional step (notshown) of illuminating the object with light from a light source (e.g.,light source 632 of FIG. 6 or light source 1032 of FIGS. 10A-11, 12A,and 13A-18). In some embodiments, the light may comprise range ofwavelengths including IR light.

In some embodiments, the off-axis image produced at block 1930 may beprocessed and analyzed, for example, using image-processing techniques.The analyzed off-axis image may be used to perform one or more of: eyetracking; biometric identification; multiscopic reconstruction of ashape of an eye; estimating an accommodation state of an eye; or imaginga retina, iris, other distinguishing pattern of an eye, and evaluate aphysiological state of the user based, in part, on the analyzed off-axisimage, as described above and throughout this disclosure.

In various embodiments, the routine 1900 may be performed by a hardwareprocessor (e.g., the local processing and data module 140 of FIG. 2)configured to execute instructions stored in a memory. In otherembodiments, a remote computing device (in network communication withthe display apparatus) with computer-executable instructions can causethe display apparatus to perform aspects of the routine 1900.

Additional Aspects

1. An optical device comprising: a substrate having a proximal surfaceand a distal surface; a first coupling optical element disposed on oneof the proximal surface and the distal surface; and a second couplingoptical element disposed on one of the proximal surface and the distalsurface and laterally offset from the first coupling optical elementalong a direction parallel to the proximal surface or the distalsurface, wherein the first coupling optical element is configured todeflect light at an angle to totally internally reflect (TIR) the lightbetween the proximal and distal surfaces and toward the second couplingoptical element, the second coupling optical element configured todeflect light at an angle out of the substrate.

2. The optical device of Aspect 1, wherein the substrate is transparentto visible light.

3. The optical device of Aspect 1 or 2, wherein the substrate comprisesa polymer.

4. The optical device of any one of Aspects 1-3, wherein the substratecomprises polycarbonate.

5. The optical device of any one of Aspects 1-4, wherein the first andsecond coupling optical elements are external to and fixed to at leastone of the proximal and distal surfaces of the substrate.

6. The optical device of any one of Aspects 1-5, wherein the first andsecond coupling optical elements comprise a portion of the substrate.

7. The optical device of any one of Aspects 1-6, wherein at least one ofthe first and second coupling optical elements comprise a plurality ofdiffractive features.

8. The optical device of Aspect 7, wherein the plurality of diffractivefeatures have a relatively high diffraction efficiency for a range ofwavelengths so as to diffract substantially all of the light of therange of wavelengths.

9. The optical device of Aspect 7 or 8, wherein the plurality ofdiffractive features diffract light in at least one direction based inpart on a period of the plurality of diffractive elements, wherein theat least one direction is selected to TIR the light between the proximaland distal surfaces.

10. The optical device of any one of Aspects 1-7, wherein at least oneof the first or second coupling optical elements comprises at least oneof an off-axis diffractive optical element (DOE), an off-axisdiffraction grating, an off-axis diffractive optical element (DOE), anoff-axis holographic mirror (OAHM), or an off-axis volumetricdiffractive optical element (OAVDOE), or an off-axis cholesteric liquidcrystal diffraction grating (OACLCG).

11. The optical device of any one of Aspects 1-7 and 10, wherein each ofthe first and second coupling optical elements are configured to deflectlight of a first range of wavelengths while transmitting light of asecond range of wavelengths.

12. The optical device of Aspect 11, wherein the first range ofwavelengths comprises light in at least one of the infrared (IR) ornear-IR spectrum and the second range of wavelengths comprises light inthe visible spectrum.

13. The optical device of any one of Aspects 1, 7, and 11, wherein thefirst and second coupling optical elements selectively reflect light ofa range of wavelengths, wherein the first coupling optical element isdisposed on the distal surface of the substrate and the second couplingoptical element is disposed on the proximal surface of the substrate.

14. The optical device of any one of Aspects 1, 7, 10, and 11, whereinthe first and second coupling optical elements selectively transmitlight of a range of wavelengths, wherein the first coupling opticalelement is disposed on the proximal surface of the substrate and thesecond coupling optical element is disposed on the distal surface of thesubstrate.

15. The optical device of any one of Aspects 1, 7, 10, and 11, whereinthe first coupling optical element selectively reflects light of a rangeof wavelengths and the second coupling optical element selectivelytransmits light of the range of wavelengths, wherein the first andsecond coupling optical elements are disposed on the distal surface ofthe substrate.

16. The optical device of any one of Aspects 1, 7, 10, and 11, whereinthe first coupling optical element selectively transmits light of arange of wavelengths and the second coupling optical element selectivelyreflects light of the range of wavelengths, wherein the first and secondcoupling optical elements are disposed on the proximal surface of thesubstrate.

17. A head mounted display (HMD) configured to be worn on a head of auser, the HMD comprising: a frame; a pair of optical elements supportedby the frame such that each optical element of the pair of opticalelements is capable of being disposed forward of an eye of the user; andan imaging system comprising: a camera assembly mounted to the frame;and an optical device in accordance to any one of the Aspects 1-16.

18. The HMD of Aspect 17, wherein at least one optical element of thepair of optical elements comprises the substrate.

19. The HMD of Aspect 17 or 18, wherein the substrate is disposed on asurface of at least one optical element of the pair of optical elements.

20. The HMD of any one of Aspects 17-19, wherein the frame comprises apair of ear stems, and the camera assembly is mounted on one of the pairof ear stems.

21. The HMD of any one of Aspects 17-20, wherein the camera assembly isa forward facing camera assembly configured to image light received fromthe second coupling optical element.

22. The HMD of any one of Aspects 17-20, wherein the camera assembly isa backward facing camera assembly disposed in a direction facing towardthe user, the backward facing camera assembly configured to image lightreceived from the second coupling optical element.

23. The HMD of any one of Aspects 17-22, further comprising a lightsource emitting light of a first range of wavelengths toward at leastone of: the eye of the user, a part of the eye, or a portion of tissuesurrounding the eye.

24. The HMD of Aspect 23, wherein the light of the first range ofwavelengths is reflected toward the first coupling optical element by atleast one of: the eye of the user, a part of the eye, or a portion oftissue surrounding the eye.

25. The HMD of any one of Aspects 17-23, wherein each of the pair ofoptical elements is transparent to visible light.

26. The HMD of any one of Aspects 17-23 and 25, wherein each of the pairof optical elements is configured to display an image to the user.

27. The HMD of any one of Aspects 17-23, 25, and 26, wherein cameraassembly is configured to image at least one of: the eye of the user, apart of the eye, or a portion of tissue surrounding the eye based, inpart on, light received from the second coupling optical element.

28. The HMD of Aspect 27, wherein the HMD is configured to track thegaze of the user based on the image of the at least one of the: eye ofthe user, the part of the eye, or the portion of tissue surrounding theeye.

29. The HMD of Aspect 27, wherein the image imaged by the cameraassembly is consistent with an image imaged by a camera placed in frontof the eye of the user and directly viewing the at least one of the: eyeof the user, the part of the eye, or the portion of tissue surroundingthe eye.

30. The HMD of any one of Aspects 17-23, 25, and 27, wherein the opticaldevice is arranged to reduce stray light received by the cameraassembly.

31. The HMD of any one of Aspects 17-23, 25, 27, and 30, wherein a sizeof the first coupling optical element is less than a stride distance ofthe light reflected in the between the distal and proximal surfaces ofthe substrate, wherein the stride distance is based on a thickness ofthe substrate and the angle at which the first coupling optical elementdeflects the light.

32. The HMD of Aspect 31, wherein the size of the first coupling opticalelement is based on the field of view of the eye of the user.

33. The HMD of any one of Aspects 17-23, 25, 27, 30, and 31, wherein animage of the eye of the user imaged by the camera assembly and an imageof the eye of the user imaged by a camera placed in front of the eye ofthe user are indistinguishable.

34. The HMD of any one of Aspects 17-23, 25, 27, 30, 31, and 33, furthercomprising: a non-transitory data storage configured to store imageryacquired by the camera assembly; and a hardware processor incommunication with the non-transitory data storage, the hardwareprocessor programmed with executable instructions to analyze theimagery, and perform one or more of: eye tracking; biometricidentification; multiscopic reconstruction of a shape of an eye;estimating an accommodation state of an eye; or imaging a retina, iris,other distinguishing pattern of an eye, and evaluate a physiologicalstate of the user.

35. An imaging system comprising: a substrate having a proximal surfaceand a distal surface, the substrate comprising: a first diffractiveoptical element disposed on one of the proximal surface and the distalsurface; and a second diffractive optical element disposed on one of theproximal surface and the distal surface, the second diffractive opticalelement offset from the first diffractive optical element along adirection parallel to the proximal surface or the distal surface,wherein the first diffractive optical element is configured to deflectlight at an angle to totally internally reflect (TIR) the light betweenthe proximal and distal surfaces and toward the second coupling opticalelement, the second diffractive optical element configured to deflectlight incident thereon at an angle out of the substrate; and a cameraassembly to image the light deflected by the second diffractive opticalelement.

36. The imaging system of Aspect 35, wherein the first and seconddiffractive optical elements comprise at least one of an off-axisdiffractive optical element (DOE), an off-axis diffraction grating, anoff-axis diffractive optical element (DOE), an off-axis holographicmirror (OAHM), or an off-axis volumetric diffractive optical element(OAVDOE), an off-axis cholesteric liquid crystal diffraction grating(OACLCG), a hot mirror, a prism, or a surface of a decorative lens.

37. A method of imaging an object using a virtual camera, the methodcomprises: providing an imaging system in front of an object to beimaged, wherein the imaging system comprises: a substrate comprising afirst coupling optical element and a second coupling optical elementeach disposed on one of a proximal surface and a distal surface of thesubstrate and offset from each other, wherein the first coupling opticalelement is configured to deflect light at an angle to totally internallyreflect (TIR) the light between the proximal and distal surfaces andtoward the second coupling optical element, the second coupling opticalelement configured to deflect the light at an angle out of thesubstrate; and capturing the light with a camera assembly oriented toreceive light deflected by the second coupling optical element; andproducing an off-axis image of the object based on the captured light.

38. The method of Aspect 37, wherein each of the first and secondcoupling optical elements deflect light of a first range of wavelengthswhile transmitting light in a second range of wavelengths.

39. The method of Aspect 37 or 38, further comprising illuminating theobject with a first range of wavelengths emitted by a light source.

40. The method of any one of Aspects 37-39, further comprising:analyzing the off-axis image, and performing one or more of: eyetracking; biometric identification; multiscopic reconstruction of ashape of an eye; estimating an accommodation state of an eye; or imaginga retina, iris, other distinguishing pattern of an eye, and evaluate aphysiological state of the user based, in part, on the analyzed off-axisimage.

41. An imaging system comprising: a substrate having a proximal surfaceand a distal surface; a reflective optical element adjacent to thedistal surface, wherein the reflective optical element is configured toreflect, at an angle, light that has passed out of the substrate at thedistal surface; and a camera assembly to image the light reflected bythe reflective optical element.

42. The imaging system of Aspect 41, wherein the reflective opticalelement comprises a surface of a decorative lens.

43. The imaging system of Aspect 41 or Aspect 42, wherein the reflectiveoptical element comprises a reflective coating on a surface of adecorative lens.

44. The imaging system of any one of Aspects 41-43, wherein thereflective optical element comprises a reflective prism.

45. The imaging system of any one of Aspects 41-44, wherein thereflective optical element is reflective to infrared light andtransmissive to visible light

46. The imaging system of any one of Aspects 41-45, further comprising adiffractive optical element adjacent to the proximal surface.

47. The imaging system of any one of Aspects 41-46, wherein the cameraassembly is a forward facing camera assembly configured to image lightreceived from the reflective optical element.

48. A head mounted display (HMD) configured to be worn on a head of auser, the HMD comprising: a frame; a pair of optical elements supportedby the frame such that each optical element of the pair of opticalelements is capable of being disposed forward of an eye of the user; andan imaging system in accordance with any one of claims 41-47.

49. The HMD of Aspect 48, wherein at least one optical element of thepair of optical elements comprises the substrate.

50. The HMD of Aspect 48 or 49, wherein the substrate is disposed on asurface of at least one optical element of the pair of optical elements.

51. The HMD of any one of Aspects 48-50, wherein the frame comprises apair of ear stems, and the camera assembly is mounted on one of the pairof ear stems.

52. The HMD of any one of Aspects 48-51, further comprising a lightsource emitting light of a first range of wavelengths toward at leastone of: the eye of the user, a part of the eye, or a portion of tissuesurrounding the eye.

53. The HMD of Aspect any one of Aspects 48-52, wherein each of the pairof optical elements is transparent to visible light.

54. The HMD of any one of Aspects 48-53, wherein each of the pair ofoptical elements is configured to display an image to the user.

55. The HMD of any one of Aspects 48-54, wherein the camera assembly isconfigured to image at least one of: the eye of the user, a part of theeye, or a portion of tissue surrounding the eye based, in part on, lightreceived from the second coupling optical element.

56. The HMD of any one of Aspects 48-55, wherein the HMD is configuredto track the gaze of the user based on the image of the at least one ofthe: eye of the user, the part of the eye, or the portion of tissuesurrounding the eye.

57. The HMD of any one of Aspects 48-56, wherein the image imaged by thecamera assembly is consistent with an image imaged by a camera placed infront of the eye of the user and directly viewing the at least one ofthe: eye of the user, the part of the eye, or the portion of tissuesurrounding the eye.

58. The HMD of any one of Aspects 48-57, wherein the optical device isarranged to reduce stray light received by the camera assembly.

59. The HMD of any one of Aspects 48-58, wherein an image of the eye ofthe user imaged by the camera assembly and an image of the eye of theuser imaged by a camera placed in front of the eye of the user areindistinguishable.

60. The HMD of any one of Aspects 48-59, further comprising: anon-transitory data storage configured to store imagery acquired by thecamera assembly; and a hardware processor in communication with thenon-transitory data storage, the hardware processor programmed withexecutable instructions to analyze the imagery, and perform one or moreof: eye tracking; biometric identification; multiscopic reconstructionof a shape of an eye; estimating an accommodation state of an eye; orimaging a retina, iris, other distinguishing pattern of an eye, andevaluate a physiological state of the user.

Additional Considerations

In the embodiments described above, the optical arrangements have beendescribed in the context of eye-imaging display systems and, moreparticularly, augmented reality display systems. It will be understood,however, that the principles and advantages of the optical arrangementscan be used for other head-mounted display, optical systems, apparatus,or methods. In the foregoing, it will be appreciated that any feature ofany one of the embodiments can be combined and/or substituted with anyother feature of any other one of the embodiments.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,”“include,” “including,” “have” and “having” and the like are to beconstrued in an inclusive sense, as opposed to an exclusive orexhaustive sense; that is to say, in the sense of “including, but notlimited to.” The word “coupled”, as generally used herein, refers to twoor more elements that may be either directly connected, or connected byway of one or more intermediate elements. Likewise, the word“connected”, as generally used herein, refers to two or more elementsthat may be either directly connected, or connected by way of one ormore intermediate elements. Depending on the context, “coupled” or“connected” may refer to an optical coupling or optical connection suchthat light is coupled or connected from one optical element to anotheroptical element. Additionally, the words “herein,” “above,” “below,”“infra,” “supra,” and words of similar import, when used in thisapplication, shall refer to this application as a whole and not to anyparticular portions of this application. Where the context permits,words in the above Detailed Description using the singular or pluralnumber may also include the plural or singular number, respectively. Theword “or” in reference to a list of two or more items is an inclusive(rather than an exclusive) “or”, and “or” covers all of the followinginterpretations of the word: any of the items in the list, all of theitems in the list, and any combination of one or more of the items inthe list, and does not exclude other items being added to the list. Inaddition, the articles “a,” “an,” and “the” as used in this applicationand the appended claims are to be construed to mean “one or more” or “atleast one” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Moreover, conditional language used herein, such as, among others,“can,” “could,” “might,” “may,” “e.g.,” “for example,” “such as” and thelike, unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or states. Thus, such conditional language is notgenerally intended to imply that features, elements and/or states are inany way required for one or more embodiments or whether these features,elements and/or states are included or are to be performed in anyparticular embodiment.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the disclosure. Indeed, the novel apparatus, methods, andsystems described herein may be embodied in a variety of other forms;furthermore, various omissions, substitutions and changes in the form ofthe methods and systems described herein may be made without departingfrom the spirit of the disclosure. For example, while blocks arepresented in a given arrangement, alternative embodiments may performsimilar functionalities with different components and/or circuittopologies, and some blocks may be deleted, moved, added, subdivided,combined, and/or modified. Each of these blocks may be implemented in avariety of different ways. Any suitable combination of the elements andacts of the various embodiments described above can be combined toprovide further embodiments. The various features and processesdescribed above may be implemented independently of one another, or maybe combined in various ways. No element or combinations of elements isnecessary or indispensable for all embodiments. All suitablecombinations and subcombinations of features of this disclosure areintended to fall within the scope of this disclosure.

What is claimed is:
 1. An optical device comprising: a substrate havinga proximal surface and a distal surface; a first coupling opticalelement disposed on one of the proximal surface and the distal surface;and a second coupling optical element disposed on one of the proximalsurface and the distal surface and laterally offset from the firstcoupling optical element along a direction parallel to the proximalsurface or the distal surface, wherein the first coupling opticalelement is configured to deflect light at an angle to totally internallyreflect (TIR) the light between the proximal and distal surfaces andtoward the second coupling optical element, the second coupling opticalelement configured to deflect light at an angle out of the substrate. 2.The optical device of claim 1, wherein the substrate is transparent tovisible light.
 3. The optical device of claim 1, wherein the substratecomprises a polymer.
 4. The optical device of claim 1, wherein thesubstrate comprises polycarbonate.
 5. The optical device of claim 1,wherein the first and second coupling optical elements are external toand fixed to at least one of the proximal and distal surfaces of thesubstrate.
 6. The optical device of claim 1, wherein the first andsecond coupling optical elements comprise a portion of the substrate. 7.The optical device of claim 1, wherein at least one of the first andsecond coupling optical elements comprise a plurality of diffractivefeatures.
 8. The optical device of claim 7, wherein the plurality ofdiffractive features have a relatively high diffraction efficiency for arange of wavelengths so as to diffract substantially all of the light ofthe range of wavelengths.
 9. The optical device of claim 7, wherein theplurality of diffractive features diffract light in at least onedirection based in part on a period of the plurality of diffractiveelements, wherein the at least one direction is selected to TIR thelight between the proximal and distal surfaces.
 10. The optical deviceof claim 1, wherein at least one of the first or second coupling opticalelements comprises at least one of an off-axis diffractive opticalelement (DOE), an off-axis diffraction grating, an off-axis diffractiveoptical element (DOE), an off-axis holographic mirror (OAHM), or anoff-axis volumetric diffractive optical element (OAVDOE), or an off-axischolesteric liquid crystal diffraction grating (OACLCG).
 11. The opticaldevice of claim 1, wherein each of the first and second coupling opticalelements are configured to deflect light of a first range of wavelengthswhile transmitting light of a second range of wavelengths.
 12. Theoptical device of claim 11, wherein the first range of wavelengthscomprises light in at least one of the infrared (IR) or near-IR spectrumand the second range of wavelengths comprises light in the visiblespectrum.
 13. The optical device of claim 1, wherein the first andsecond coupling optical elements selectively reflect light of a range ofwavelengths, wherein the first coupling optical element is disposed onthe distal surface of the substrate and the second coupling opticalelement is disposed on the proximal surface of the substrate.
 14. Theoptical device of claim 1, wherein the first and second coupling opticalelements selectively transmit light of a range of wavelengths, whereinthe first coupling optical element is disposed on the proximal surfaceof the substrate and the second coupling optical element is disposed onthe distal surface of the substrate.
 15. The optical device of claim 1,wherein the first coupling optical element selectively reflects light ofa range of wavelengths and the second coupling optical elementselectively transmits light of the range of wavelengths, wherein thefirst and second coupling optical elements are disposed on the distalsurface of the substrate.
 16. The optical device of claim 1, wherein thefirst coupling optical element selectively transmits light of a range ofwavelengths and the second coupling optical element selectively reflectslight of the range of wavelengths, wherein the first and second couplingoptical elements are disposed on the proximal surface of the substrate.17. A head mounted display (HMD) configured to be worn on a head of auser, the HMD comprising: a frame; a pair of optical elements supportedby the frame such that each optical element of the pair of opticalelements is capable of being disposed forward of an eye of the user; andan imaging system comprising: a camera assembly mounted to the frame;and an optical device in accordance with claim
 1. 18. The HMD of claim17, wherein at least one optical element of the pair of optical elementscomprises the substrate.
 19. The HMD of claim 17, wherein the substrateis disposed on a surface of at least one optical element of the pair ofoptical elements.
 20. The HMD of claim 17, wherein the frame comprises apair of ear stems, and the camera assembly is mounted on one of the pairof ear stems.
 21. The HMD of claim 17, wherein the camera assembly is aforward facing camera assembly configured to image light received fromthe second coupling optical element.
 22. The HMD of claim 17, whereinthe camera assembly is a backward facing camera assembly disposed in adirection facing toward the user, the backward facing camera assemblyconfigured to image light received from the second coupling opticalelement.
 23. The HMD of claim 17, further comprising a light sourceemitting light of a first range of wavelengths toward at least one of:the eye of the user, a part of the eye, or a portion of tissuesurrounding the eye.
 24. The HMD of claim 23, wherein the light of thefirst range of wavelengths is reflected toward the first couplingoptical element by at least one of: the eye of the user, a part of theeye, or a portion of tissue surrounding the eye.
 25. The HMD of claim17, wherein each of the pair of optical elements is transparent tovisible light.
 26. The HMD of claim 17, wherein each of the pair ofoptical elements is configured to display an image to the user.
 27. TheHMD of claim 17, wherein camera assembly is configured to image at leastone of: the eye of the user, a part of the eye, or a portion of tissuesurrounding the eye based, in part on, light received from the secondcoupling optical element.
 28. The HMD of claim 27, wherein the HMD isconfigured to track the gaze of the user based on the image of the atleast one of the: eye of the user, the part of the eye, or the portionof tissue surrounding the eye.
 29. The HMD of claim 27, wherein theimage imaged by the camera assembly is consistent with an image imagedby a camera placed in front of the eye of the user and directly viewingthe at least one of the: eye of the user, the part of the eye, or theportion of tissue surrounding the eye.
 30. The HMD of claim 17, whereinthe optical device is arranged to reduce stray light received by thecamera assembly.
 31. The HMD of claim 17, wherein a size of the firstcoupling optical element is less than a stride distance of the lightreflected in the between the distal and proximal surfaces of thesubstrate, wherein the stride distance is based on a thickness of thesubstrate and the angle at which the first coupling optical elementdeflects the light.
 32. The HMD of claim 31, wherein the size of thefirst coupling optical element is based on the field of view of the eyeof the user.
 33. The HMD of claim 17, wherein an image of the eye of theuser imaged by the camera assembly and an image of the eye of the userimaged by a camera placed in front of the eye of the user areindistinguishable.
 34. The HMD of claim 17, further comprising: anon-transitory data storage configured to store imagery acquired by thecamera assembly; and a hardware processor in communication with thenon-transitory data storage, the hardware processor programmed withexecutable instructions to analyze the imagery, and perform one or moreof: eye tracking; biometric identification; multiscopic reconstructionof a shape of an eye; estimating an accommodation state of an eye; orimaging a retina, iris, other distinguishing pattern of an eye, andevaluate a physiological state of the user.
 35. An imaging systemcomprising: a substrate having a proximal surface and a distal surface,the substrate comprising: a first diffractive optical element disposedon one of the proximal surface and the distal surface; and a seconddiffractive optical element disposed on one of the proximal surface andthe distal surface, the second diffractive optical element offset fromthe first diffractive optical element along a direction parallel to theproximal surface or the distal surface, wherein the first diffractiveoptical element is configured to deflect light at an angle to totallyinternally reflect (TIR) the light between the proximal and distalsurfaces and toward the second coupling optical element, the seconddiffractive optical element configured to deflect light incident thereonat an angle out of the substrate; and a camera assembly to image thelight deflected by the second diffractive optical element.
 36. Theimaging system of claim 35, wherein the first and second diffractiveoptical elements comprise at least one of an off-axis diffractiveoptical element (DOE), an off-axis diffraction grating, an off-axisdiffractive optical element (DOE), an off-axis holographic mirror(OAHM), or an off-axis volumetric diffractive optical element (OAVDOE),an off-axis cholesteric liquid crystal diffraction grating (OACLCG), ahot mirror, a prism, or a surface of a decorative lens.
 37. A method ofimaging an object using a virtual camera, the method comprises:providing an imaging system in front of an object to be imaged, whereinthe imaging system comprises: a substrate comprising a first couplingoptical element and a second coupling optical element each disposed onone of a proximal surface and a distal surface of the substrate andoffset from each other, wherein the first coupling optical element isconfigured to deflect light at an angle to totally internally reflect(TIR) the light between the proximal and distal surfaces and toward thesecond coupling optical element, the second coupling optical elementconfigured to deflect the light at an angle out of the substrate; andcapturing the light with a camera assembly oriented to receive lightdeflected by the second coupling optical element; and producing anoff-axis image of the object based on the captured light.
 38. The methodof claim 37, wherein each of the first and second coupling opticalelements deflect light of a first range of wavelengths whiletransmitting light in a second range of wavelengths.
 39. The method ofclaim 37, further comprising illuminating the object with a first rangeof wavelengths emitted by a light source.
 40. The method of claim 37,further comprising: analyzing the off-axis image, and performing one ormore of: eye tracking; biometric identification; multiscopicreconstruction of a shape of an eye; estimating an accommodation stateof an eye; or imaging a retina, iris, other distinguishing pattern of aneye, and evaluate a physiological state of the user based, in part, onthe analyzed off-axis image.